{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c024bfa4-1a7a-4751-b5a1-827225a3478b",
   "metadata": {
    "id": "c024bfa4-1a7a-4751-b5a1-827225a3478b"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfabadb8-5935-45ff-b39c-db7a29012129",
   "metadata": {
    "id": "bfabadb8-5935-45ff-b39c-db7a29012129"
   },
   "source": [
    "# Chapter 6: Finetuning for Text Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
    "outputId": "9495f150-9d79-4910-d6e7-6c0d9aae4a41"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matplotlib version: 3.10.7\n",
      "numpy version: 2.3.4\n",
      "tiktoken version: 0.12.0\n",
      "torch version: 2.9.0\n",
      "tensorflow version: 2.20.0\n",
      "pandas version: 2.3.3\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"matplotlib\",  # Plotting library\n",
    "        \"numpy\",       # PyTorch & TensorFlow dependency\n",
    "        \"tiktoken\",    # Tokenizer\n",
    "        \"torch\",       # Deep learning library\n",
    "        \"tensorflow\",  # For OpenAI's pretrained weights\n",
    "        \"pandas\"       # Dataset loading\n",
    "       ]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a445828a-ff10-4efa-9f60-a2e2aed4c87d",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/01.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c",
   "metadata": {
    "id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c"
   },
   "source": [
    "## 6.1 Different categories of finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ede3d731-5123-4f02-accd-c670ce50a5a3",
   "metadata": {
    "id": "ede3d731-5123-4f02-accd-c670ce50a5a3"
   },
   "source": [
    "- No code in this section"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac45579d-d485-47dc-829e-43be7f4db57b",
   "metadata": {},
   "source": [
    "- The most common ways to finetune language models are instruction-finetuning and classification finetuning\n",
    "- Instruction-finetuning, depicted below, is the topic of the next chapter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c29ef42-46d9-43d4-8bb4-94974e1665e4",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/02.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f60321-95b8-46a9-97bf-1d07fda2c3dd",
   "metadata": {},
   "source": [
    "- Classification finetuning, the topic of this chapter, is a procedure you may already be familiar with if you have a background in machine learning -- it's similar to training a convolutional network to classify handwritten digits, for example\n",
    "- In classification finetuning, we have a specific number of class labels (for example, \"spam\" and \"not spam\") that the model can output\n",
    "- A classification finetuned model can only predict classes it has seen during training (for example, \"spam\" or \"not spam\"), whereas an instruction-finetuned model can usually perform many tasks\n",
    "- We can think of a classification-finetuned model as a very specialized model; in practice, it is much easier to create a specialized model than a generalist model that performs well on many different tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b37a0c4-0bb1-4061-b1fe-eaa4416d52c3",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/03.webp\" width=400px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf",
   "metadata": {
    "id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf"
   },
   "source": [
    "## 6.2 Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f628975-d2e8-4f7f-ab38-92bb868b7067",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/04.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbd459f-63fa-4d8c-8499-e23103156c7d",
   "metadata": {
    "id": "9fbd459f-63fa-4d8c-8499-e23103156c7d"
   },
   "source": [
    "- This section prepares the dataset we use for classification finetuning\n",
    "- We use a dataset consisting of spam and non-spam text messages to finetune the LLM to classify them\n",
    "- First, we download and unzip the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
    "outputId": "424e4423-f623-443c-ab9e-656f9e867559"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\nimport urllib.request\\nimport zipfile\\nimport os\\nfrom pathlib import Path\\n\\nurl = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\\nzip_path = \"sms_spam_collection.zip\"\\nextracted_path = \"sms_spam_collection\"\\ndata_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\\n\\ndef download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\\n    if data_file_path.exists():\\n        print(f\"{data_file_path} already exists. Skipping download and extraction.\")\\n        return\\n\\n    # Downloading the file\\n    with urllib.request.urlopen(url) as response:\\n        with open(zip_path, \"wb\") as out_file:\\n            out_file.write(response.read())\\n\\n    # Unzipping the file\\n    with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\\n        zip_ref.extractall(extracted_path)\\n\\n    # Add .tsv file extension\\n    original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\\n    os.rename(original_file_path, data_file_path)\\n    print(f\"File downloaded and saved as {data_file_path}\")\\n\\ntry:\\n    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\\nexcept (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as e:\\n    print(f\"Primary URL failed: {e}. Trying backup URL...\")\\n    url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\\n    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\\n'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "import zipfile\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
    "zip_path = \"sms_spam_collection.zip\"\n",
    "extracted_path = \"sms_spam_collection\"\n",
    "data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
    "\n",
    "\n",
    "def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
    "    if data_file_path.exists():\n",
    "        print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
    "        return\n",
    "\n",
    "    # Downloading the file\n",
    "    response = requests.get(url, stream=True, timeout=60)\n",
    "    response.raise_for_status()\n",
    "    with open(zip_path, \"wb\") as out_file:\n",
    "        for chunk in response.iter_content(chunk_size=8192):\n",
    "            if chunk:\n",
    "                out_file.write(chunk)\n",
    "\n",
    "    # Unzipping the file\n",
    "    with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(extracted_path)\n",
    "\n",
    "    # Add .tsv file extension\n",
    "    original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
    "    os.rename(original_file_path, data_file_path)\n",
    "    print(f\"File downloaded and saved as {data_file_path}\")\n",
    "\n",
    "\n",
    "try:\n",
    "    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
    "except (requests.exceptions.RequestException, TimeoutError) as e:\n",
    "    print(f\"Primary URL failed: {e}. Trying backup URL...\")\n",
    "    url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\n",
    "    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
    "\n",
    "\n",
    "\n",
    "# The book originally used the following code below\n",
    "# However, urllib uses older protocol settings that\n",
    "# can cause problems for some readers using a VPN.\n",
    "# The `requests` version above is more robust\n",
    "# in that regard.\n",
    "\n",
    "\"\"\"\n",
    "import urllib.request\n",
    "import zipfile\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
    "zip_path = \"sms_spam_collection.zip\"\n",
    "extracted_path = \"sms_spam_collection\"\n",
    "data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
    "\n",
    "def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
    "    if data_file_path.exists():\n",
    "        print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
    "        return\n",
    "\n",
    "    # Downloading the file\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        with open(zip_path, \"wb\") as out_file:\n",
    "            out_file.write(response.read())\n",
    "\n",
    "    # Unzipping the file\n",
    "    with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(extracted_path)\n",
    "\n",
    "    # Add .tsv file extension\n",
    "    original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
    "    os.rename(original_file_path, data_file_path)\n",
    "    print(f\"File downloaded and saved as {data_file_path}\")\n",
    "\n",
    "try:\n",
    "    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
    "except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as e:\n",
    "    print(f\"Primary URL failed: {e}. Trying backup URL...\")\n",
    "    url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\n",
    "    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1",
   "metadata": {
    "id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1"
   },
   "source": [
    "- The dataset is saved as a tab-separated text file, which we can load into a pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
    "outputId": "a16c5cde-d341-4887-a93f-baa9bec542ab"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>spam</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>ham</td>\n",
       "      <td>Will ü b going to esplanade fr home?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5569</th>\n",
       "      <td>ham</td>\n",
       "      <td>Pity, * was in mood for that. So...any other s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5570</th>\n",
       "      <td>ham</td>\n",
       "      <td>The guy did some bitching but I acted like i'd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5571</th>\n",
       "      <td>ham</td>\n",
       "      <td>Rofl. Its true to its name</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5572 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Label                                               Text\n",
       "0      ham  Go until jurong point, crazy.. Available only ...\n",
       "1      ham                      Ok lar... Joking wif u oni...\n",
       "2     spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3      ham  U dun say so early hor... U c already then say...\n",
       "4      ham  Nah I don't think he goes to usf, he lives aro...\n",
       "...    ...                                                ...\n",
       "5567  spam  This is the 2nd time we have tried 2 contact u...\n",
       "5568   ham               Will ü b going to esplanade fr home?\n",
       "5569   ham  Pity, * was in mood for that. So...any other s...\n",
       "5570   ham  The guy did some bitching but I acted like i'd...\n",
       "5571   ham                         Rofl. Its true to its name\n",
       "\n",
       "[5572 rows x 2 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(data_file_path, sep=\"\\t\", header=None, names=[\"Label\", \"Text\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b6e631-4f0b-4aab-82b9-8898e6663109",
   "metadata": {
    "id": "e7b6e631-4f0b-4aab-82b9-8898e6663109"
   },
   "source": [
    "- When we check the class distribution, we see that the data contains \"ham\" (i.e., \"not spam\") much more frequently than \"spam\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
    "outputId": "761e0482-43ba-4f46-f4b7-6774dae51b38"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     4825\n",
      "spam     747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f773f054-0bdc-4aad-bbf6-397621bf63db",
   "metadata": {
    "id": "f773f054-0bdc-4aad-bbf6-397621bf63db"
   },
   "source": [
    "- For simplicity, and because we prefer a small dataset for educational purposes anyway (it will make it possible to finetune the LLM faster), we subsample (undersample) the dataset so that it contains 747 instances from each class\n",
    "- (Next to undersampling, there are several other ways to deal with class balances, but they are out of the scope of a book on LLMs; you can find examples and more information in the [`imbalanced-learn` user guide](https://imbalanced-learn.org/stable/user_guide.html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7be4a0a2-9704-4a96-b38f-240339818688",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7be4a0a2-9704-4a96-b38f-240339818688",
    "outputId": "396dc415-cb71-4a88-e85d-d88201c6d73f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     747\n",
      "spam    747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "def create_balanced_dataset(df):\n",
    "    \n",
    "    # Count the instances of \"spam\"\n",
    "    num_spam = df[df[\"Label\"] == \"spam\"].shape[0]\n",
    "    \n",
    "    # Randomly sample \"ham\" instances to match the number of \"spam\" instances\n",
    "    ham_subset = df[df[\"Label\"] == \"ham\"].sample(num_spam, random_state=123)\n",
    "    \n",
    "    # Combine ham \"subset\" with \"spam\"\n",
    "    balanced_df = pd.concat([ham_subset, df[df[\"Label\"] == \"spam\"]])\n",
    "\n",
    "    return balanced_df\n",
    "\n",
    "\n",
    "balanced_df = create_balanced_dataset(df)\n",
    "print(balanced_df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6",
   "metadata": {
    "id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6"
   },
   "source": [
    "- Next, we change the string class labels \"ham\" and \"spam\" into integer class labels 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd",
   "metadata": {
    "id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd"
   },
   "outputs": [],
   "source": [
    "balanced_df[\"Label\"] = balanced_df[\"Label\"].map({\"ham\": 0, \"spam\": 1})    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e6f7f062-ef4e-4020-8275-71990cab4414",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4307</th>\n",
       "      <td>0</td>\n",
       "      <td>Awww dat is sweet! We can think of something t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4138</th>\n",
       "      <td>0</td>\n",
       "      <td>Just got to  &amp;lt;#&amp;gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4831</th>\n",
       "      <td>0</td>\n",
       "      <td>The word \"Checkmate\" in chess comes from the P...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4461</th>\n",
       "      <td>0</td>\n",
       "      <td>This is wishing you a great day. Moji told me ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5440</th>\n",
       "      <td>0</td>\n",
       "      <td>Thank you. do you generally date the brothas?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5537</th>\n",
       "      <td>1</td>\n",
       "      <td>Want explicit SEX in 30 secs? Ring 02073162414...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5540</th>\n",
       "      <td>1</td>\n",
       "      <td>ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5547</th>\n",
       "      <td>1</td>\n",
       "      <td>Had your contract mobile 11 Mnths? Latest Moto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5566</th>\n",
       "      <td>1</td>\n",
       "      <td>REMINDER FROM O2: To get 2.50 pounds free call...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>1</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1494 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Label                                               Text\n",
       "4307      0  Awww dat is sweet! We can think of something t...\n",
       "4138      0                             Just got to  &lt;#&gt;\n",
       "4831      0  The word \"Checkmate\" in chess comes from the P...\n",
       "4461      0  This is wishing you a great day. Moji told me ...\n",
       "5440      0      Thank you. do you generally date the brothas?\n",
       "...     ...                                                ...\n",
       "5537      1  Want explicit SEX in 30 secs? Ring 02073162414...\n",
       "5540      1  ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...\n",
       "5547      1  Had your contract mobile 11 Mnths? Latest Moto...\n",
       "5566      1  REMINDER FROM O2: To get 2.50 pounds free call...\n",
       "5567      1  This is the 2nd time we have tried 2 contact u...\n",
       "\n",
       "[1494 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5715e685-35b4-4b45-a86c-8a8694de9d6f",
   "metadata": {
    "id": "5715e685-35b4-4b45-a86c-8a8694de9d6f"
   },
   "source": [
    "- Let's now define a function that randomly divides the dataset into training, validation, and test subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "uQl0Psdmx15D",
   "metadata": {
    "id": "uQl0Psdmx15D"
   },
   "outputs": [],
   "source": [
    "def random_split(df, train_frac, validation_frac):\n",
    "    # Shuffle the entire DataFrame\n",
    "    df = df.sample(frac=1, random_state=123).reset_index(drop=True)\n",
    "\n",
    "    # Calculate split indices\n",
    "    train_end = int(len(df) * train_frac)\n",
    "    validation_end = train_end + int(len(df) * validation_frac)\n",
    "\n",
    "    # Split the DataFrame\n",
    "    train_df = df[:train_end]\n",
    "    validation_df = df[train_end:validation_end]\n",
    "    test_df = df[validation_end:]\n",
    "\n",
    "    return train_df, validation_df, test_df\n",
    "\n",
    "train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)\n",
    "# Test size is implied to be 0.2 as the remainder\n",
    "\n",
    "train_df.to_csv(\"train.csv\", index=None)\n",
    "validation_df.to_csv(\"validation.csv\", index=None)\n",
    "test_df.to_csv(\"test.csv\", index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d7a0c5-1d5f-458a-b685-3f49520b0094",
   "metadata": {},
   "source": [
    "## 6.3 Creating data loaders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c",
   "metadata": {
    "id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c"
   },
   "source": [
    "- Note that the text messages have different lengths; if we want to combine multiple training examples in a batch, we have to either\n",
    "  1. truncate all messages to the length of the shortest message in the dataset or batch\n",
    "  2. pad all messages to the length of the longest message in the dataset or batch\n",
    "\n",
    "- We choose option 2 and pad all messages to the longest message in the dataset\n",
    "- For that, we use `<|endoftext|>` as a padding token, as discussed in chapter 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0829f33f-1428-4f22-9886-7fee633b3666",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/06.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
    "outputId": "b5b48439-32c8-4b37-cca2-c9dc8fa86563"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50256]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f582ff-68bf-450e-bd87-5fb61afe431c",
   "metadata": {
    "id": "04f582ff-68bf-450e-bd87-5fb61afe431c"
   },
   "source": [
    "- The `SpamDataset` class below identifies the longest sequence in the training dataset and adds the padding token to the others to match that sequence length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d7791b52-af18-4ac4-afa9-b921068e383e",
   "metadata": {
    "id": "d7791b52-af18-4ac4-afa9-b921068e383e"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class SpamDataset(Dataset):\n",
    "    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n",
    "        self.data = pd.read_csv(csv_file)\n",
    "\n",
    "        # Pre-tokenize texts\n",
    "        self.encoded_texts = [\n",
    "            tokenizer.encode(text) for text in self.data[\"Text\"]\n",
    "        ]\n",
    "\n",
    "        if max_length is None:\n",
    "            self.max_length = self._longest_encoded_length()\n",
    "        else:\n",
    "            self.max_length = max_length\n",
    "            # Truncate sequences if they are longer than max_length\n",
    "            self.encoded_texts = [\n",
    "                encoded_text[:self.max_length]\n",
    "                for encoded_text in self.encoded_texts\n",
    "            ]\n",
    "\n",
    "        # Pad sequences to the longest sequence\n",
    "        self.encoded_texts = [\n",
    "            encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))\n",
    "            for encoded_text in self.encoded_texts\n",
    "        ]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        encoded = self.encoded_texts[index]\n",
    "        label = self.data.iloc[index][\"Label\"]\n",
    "        return (\n",
    "            torch.tensor(encoded, dtype=torch.long),\n",
    "            torch.tensor(label, dtype=torch.long)\n",
    "        )\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def _longest_encoded_length(self):\n",
    "        max_length = 0\n",
    "        for encoded_text in self.encoded_texts:\n",
    "            encoded_length = len(encoded_text)\n",
    "            if encoded_length > max_length:\n",
    "                max_length = encoded_length\n",
    "        return max_length\n",
    "        # Note: A more pythonic version to implement this method\n",
    "        # is the following, which is also used in the next chapter:\n",
    "        # return max(len(encoded_text) for encoded_text in self.encoded_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "uzj85f8ou82h",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uzj85f8ou82h",
    "outputId": "d08f1cf0-c24d-445f-a3f8-793532c3716f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\n"
     ]
    }
   ],
   "source": [
    "train_dataset = SpamDataset(\n",
    "    csv_file=\"train.csv\",\n",
    "    max_length=None,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "\n",
    "print(train_dataset.max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15bdd932-97eb-4b88-9cf9-d766ea4c3a60",
   "metadata": {},
   "source": [
    "- We also pad the validation and test set to the longest training sequence\n",
    "- Note that validation and test set samples that are longer than the longest training example are being truncated via `encoded_text[:self.max_length]` in the `SpamDataset` code\n",
    "- This behavior is entirely optional, and it would also work well if we set `max_length=None` in both the validation and test set cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e",
   "metadata": {
    "id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e"
   },
   "outputs": [],
   "source": [
    "val_dataset = SpamDataset(\n",
    "    csv_file=\"validation.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "test_dataset = SpamDataset(\n",
    "    csv_file=\"test.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20170d89-85a0-4844-9887-832f5d23432a",
   "metadata": {},
   "source": [
    "- Next, we use the dataset to instantiate the data loaders, which is similar to creating the data loaders in previous chapters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64bcc349-205f-48f8-9655-95ff21f5e72f",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/07.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
    "outputId": "3266c410-4fdb-4a8c-a142-7f707e2525ab"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "num_workers = 0\n",
    "batch_size = 8\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = DataLoader(\n",
    "    dataset=train_dataset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=True,\n",
    ")\n",
    "\n",
    "val_loader = DataLoader(\n",
    "    dataset=val_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")\n",
    "\n",
    "test_loader = DataLoader(\n",
    "    dataset=test_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7335db-e0bb-4e27-80c5-eea11e593a57",
   "metadata": {},
   "source": [
    "- As a verification step, we iterate through the data loaders and ensure that the batches contain 8 training examples each, where each training example consists of 120 tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4dee6882-4c3a-4964-af15-fa31f86ad047",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loader:\n",
      "Input batch dimensions: torch.Size([8, 120])\n",
      "Label batch dimensions torch.Size([8])\n"
     ]
    }
   ],
   "source": [
    "print(\"Train loader:\")\n",
    "for input_batch, target_batch in train_loader:\n",
    "    pass\n",
    "\n",
    "print(\"Input batch dimensions:\", input_batch.shape)\n",
    "print(\"Label batch dimensions\", target_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1",
   "metadata": {},
   "source": [
    "- Lastly, let's print the total number of batches in each dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "IZfw-TYD2zTj",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IZfw-TYD2zTj",
    "outputId": "6934bbf2-9797-4fbe-d26b-1a246e18c2fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "130 training batches\n",
      "19 validation batches\n",
      "38 test batches\n"
     ]
    }
   ],
   "source": [
    "print(f\"{len(train_loader)} training batches\")\n",
    "print(f\"{len(val_loader)} validation batches\")\n",
    "print(f\"{len(test_loader)} test batches\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c",
   "metadata": {
    "id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c"
   },
   "source": [
    "## 6.4 Initializing a model with pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e1af8b-8bd1-4b44-8b8b-dc031496e208",
   "metadata": {},
   "source": [
    "- In this section, we initialize the pretrained model we worked with in the previous chapter\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/08.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2992d779-f9fb-4812-a117-553eb790a5a9",
   "metadata": {
    "id": "2992d779-f9fb-4812-a117-553eb790a5a9"
   },
   "outputs": [],
   "source": [
    "CHOOSE_MODEL = \"gpt2-small (124M)\"\n",
    "INPUT_PROMPT = \"Every effort moves\"\n",
    "\n",
    "BASE_CONFIG = {\n",
    "    \"vocab_size\": 50257,     # Vocabulary size\n",
    "    \"context_length\": 1024,  # Context length\n",
    "    \"drop_rate\": 0.0,        # Dropout rate\n",
    "    \"qkv_bias\": True         # Query-key-value bias\n",
    "}\n",
    "\n",
    "model_configs = {\n",
    "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
    "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
    "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
    "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
    "}\n",
    "\n",
    "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
    "\n",
    "assert train_dataset.max_length <= BASE_CONFIG[\"context_length\"], (\n",
    "    f\"Dataset length {train_dataset.max_length} exceeds model's context \"\n",
    "    f\"length {BASE_CONFIG['context_length']}. Reinitialize data sets with \"\n",
    "    f\"`max_length={BASE_CONFIG['context_length']}`\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "022a649a-44f5-466c-8a8e-326c063384f5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "022a649a-44f5-466c-8a8e-326c063384f5",
    "outputId": "7091e401-8442-4f47-a1d9-ecb42a1ef930"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File already exists and is up-to-date: gpt2/124M/checkpoint\n",
      "File already exists and is up-to-date: gpt2/124M/encoder.json\n",
      "File already exists and is up-to-date: gpt2/124M/hparams.json\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.data-00000-of-00001\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.index\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.meta\n",
      "File already exists and is up-to-date: gpt2/124M/vocab.bpe\n"
     ]
    }
   ],
   "source": [
    "from gpt_download import download_and_load_gpt2\n",
    "from previous_chapters import GPTModel, load_weights_into_gpt\n",
    "# If the `previous_chapters.py` file is not available locally,\n",
    "# you can import it from the `llms-from-scratch` PyPI package.\n",
    "# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
    "# E.g.,\n",
    "# from llms_from_scratch.ch04 import GPTModel\n",
    "# from llms_from_scratch.ch05 import download_and_load_gpt2, load_weights_into_gpt\n",
    "\n",
    "model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
    "settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"gpt2\")\n",
    "\n",
    "model = GPTModel(BASE_CONFIG)\n",
    "load_weights_into_gpt(model, params)\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8e056c-abe0-415f-b34d-df686204259e",
   "metadata": {},
   "source": [
    "- To ensure that the model was loaded correctly, let's double-check that it generates coherent text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d8ac25ff-74b1-4149-8dc5-4c429d464330",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Every effort moves you forward.\n",
      "\n",
      "The first step is to understand the importance of your work\n"
     ]
    }
   ],
   "source": [
    "from previous_chapters import (\n",
    "    generate_text_simple,\n",
    "    text_to_token_ids,\n",
    "    token_ids_to_text\n",
    ")\n",
    "\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import (\n",
    "#    generate_text_simple,\n",
    "#    text_to_token_ids,\n",
    "#    token_ids_to_text\n",
    "# )\n",
    "\n",
    "\n",
    "text_1 = \"Every effort moves you\"\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_1, tokenizer),\n",
    "    max_new_tokens=15,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69162550-6a02-4ece-8db1-06c71d61946f",
   "metadata": {},
   "source": [
    "- Before we finetune the model as a classifier, let's see if the model can perhaps already classify spam messages via prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "94224aa9-c95a-4f8a-a420-76d01e3a800c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.'\n",
      "\n",
      "The following text 'spam'? Answer with 'yes' or 'no': 'You are a winner\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Is the following text 'spam'? Answer with 'yes' or 'no':\"\n",
    "    \" 'You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.'\"\n",
    ")\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_2, tokenizer),\n",
    "    max_new_tokens=23,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce39ed0-2c77-410d-8392-dd15d4b22016",
   "metadata": {},
   "source": [
    "- As we can see, the model is not very good at following instructions\n",
    "- This is expected, since it has only been pretrained and not instruction-finetuned (instruction finetuning will be covered in the next chapter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522",
   "metadata": {
    "id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522"
   },
   "source": [
    "## 6.5 Adding a classification head"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e9d66f-76b2-40fc-9ec5-3f972a8db9c0",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/09.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "217bac05-78df-4412-bd80-612f8061c01d",
   "metadata": {},
   "source": [
    "- In this section, we are modifying the pretrained LLM to make it ready for classification finetuning\n",
    "- Let's take a look at the model architecture first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b23aff91-6bd0-48da-88f6-353657e6c981",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1d8f7a01-b7c0-48d4-b1e7-8c12cc7ad932",
    "outputId": "b6a5b9b5-a92f-498f-d7cb-b58dd99e4497"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPTModel(\n",
      "  (tok_emb): Embedding(50257, 768)\n",
      "  (pos_emb): Embedding(1024, 768)\n",
      "  (drop_emb): Dropout(p=0.0, inplace=False)\n",
      "  (trf_blocks): Sequential(\n",
      "    (0): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (1): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (2): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (3): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (4): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (5): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (6): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (7): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (8): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (9): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (10): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (11): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "  )\n",
      "  (final_norm): LayerNorm()\n",
      "  (out_head): Linear(in_features=768, out_features=50257, bias=False)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f640a76-dd00-4769-9bc8-1aed0cec330d",
   "metadata": {},
   "source": [
    "- Above, we can see the architecture we implemented in chapter 4 neatly laid out\n",
    "- The goal is to replace and finetune the output layer\n",
    "- To achieve this, we first freeze the model, meaning that we make all layers non-trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fkMWFl-0etea",
   "metadata": {
    "id": "fkMWFl-0etea"
   },
   "outputs": [],
   "source": [
    "for param in model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72155f83-87d9-476a-a978-a15aa2d44147",
   "metadata": {},
   "source": [
    "- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n",
    "- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"not spam\"), we can replace the output layer as shown below, which will be trainable by default\n",
    "- Note that we use `BASE_CONFIG[\"emb_dim\"]` (which is equal to 768 in the `\"gpt2-small (124M)\"` model) to keep the code below more general"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7e759fa0-0f69-41be-b576-17e5f20e04cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "num_classes = 2\n",
    "model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30be5475-ae77-4f97-8f3e-dec462b1339f",
   "metadata": {},
   "source": [
    "- Technically, it's sufficient to only train the output layer\n",
    "- However, as I found in [Finetuning Large Language Models](https://magazine.sebastianraschka.com/p/finetuning-large-language-models), experiments show that finetuning additional layers can noticeably improve the performance\n",
    "- So, we are also making the last transformer block and the final `LayerNorm` module connecting the last transformer block to the output layer trainable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be7c1eb-c46c-4065-8525-eea1b8c66d10",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/10.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7",
   "metadata": {
    "id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7"
   },
   "outputs": [],
   "source": [
    "for param in model.trf_blocks[-1].parameters():\n",
    "    param.requires_grad = True\n",
    "\n",
    "for param in model.final_norm.parameters():\n",
    "    param.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f012b899-8284-4d3a-97c0-8a48eb33ba2e",
   "metadata": {},
   "source": [
    "- We can still use this model similar to before in previous chapters\n",
    "- For example, let's feed it some text input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
    "outputId": "27e041b1-d731-48a1-cf60-f22d4565304e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs: tensor([[5211,  345,  423,  640]])\n",
      "Inputs dimensions: torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "inputs = tokenizer.encode(\"Do you have time\")\n",
    "inputs = torch.tensor(inputs).unsqueeze(0)\n",
    "print(\"Inputs:\", inputs)\n",
    "print(\"Inputs dimensions:\", inputs.shape) # shape: (batch_size, num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbf8481-772d-467b-851c-a62b86d0cb1b",
   "metadata": {},
   "source": [
    "- What's different compared to previous chapters is that it now has two output dimensions instead of 50,257"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
    "outputId": "9cae7448-253d-4776-973e-0af190b06354"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputs:\n",
      " tensor([[[-1.5854,  0.9904],\n",
      "         [-3.7235,  7.4548],\n",
      "         [-2.2661,  6.6049],\n",
      "         [-3.5983,  3.9902]]])\n",
      "Outputs dimensions: torch.Size([1, 4, 2])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    outputs = model(inputs)\n",
    "\n",
    "print(\"Outputs:\\n\", outputs)\n",
    "print(\"Outputs dimensions:\", outputs.shape) # shape: (batch_size, num_tokens, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75430a01-ef9c-426a-aca0-664689c4f461",
   "metadata": {},
   "source": [
    "- As discussed in previous chapters, for each input token, there's one output vector\n",
    "- Since we fed the model a text sample with 4 input tokens, the output consists of 4 2-dimensional output vectors above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df9144f-6817-4be4-8d4b-5d4dadfe4a9b",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/11.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3bb8616-c791-4f5c-bac0-5302f663e46a",
   "metadata": {},
   "source": [
    "- In chapter 3, we discussed the attention mechanism, which connects each input token to each other input token\n",
    "- In chapter 3, we then also introduced the causal attention mask that is used in GPT-like models; this causal mask lets a current token only attend to the current and previous token positions\n",
    "- Based on this causal attention mechanism, the 4th (last) token contains the most information among all tokens because it's the only token that includes information about all other tokens\n",
    "- Hence, we are particularly interested in this last token, which we will finetune for the spam classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
    "outputId": "e79eb155-fa1f-46ed-ff8c-d828c3a3fabd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.5983,  3.9902]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8df08ae0-e664-4670-b7c5-8a2280d9b41b",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/12.webp\" width=200px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32aa4aef-e1e9-491b-9adf-5aa973e59b8c",
   "metadata": {},
   "source": [
    "## 6.6 Calculating the classification loss and accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "669e1fd1-ace8-44b4-b438-185ed0ba8b33",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/13.webp\" width=300px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7df4ee-0a34-4a4d-896d-affbbf81e0b3",
   "metadata": {},
   "source": [
    "- Before explaining the loss calculation, let's have a brief look at how the model outputs are turned into class labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "557996dd-4c6b-49c4-ab83-f60ef7e1d69e",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/14.webp\" width=600px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c77faab1-3461-4118-866a-6171f2b89aa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.5983,  3.9902]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edd71fa-628a-4d00-b81d-6d8bcb2c341d",
   "metadata": {},
   "source": [
    "- Similar to chapter 5, we convert the outputs (logits) into probability scores via the `softmax` function and then obtain the index position of the largest probability value via the `argmax` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b81efa92-9be1-4b9e-8790-ce1fc7b17f01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "probas = torch.softmax(outputs[:, -1, :], dim=-1)\n",
    "label = torch.argmax(probas)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "414a6f02-307e-4147-a416-14d115bf8179",
   "metadata": {},
   "source": [
    "- Note that the softmax function is optional here, as explained in chapter 5, because the largest outputs correspond to the largest probability scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f9f9ad66-4969-4501-8239-3ccdb37e71a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "logits = outputs[:, -1, :]\n",
    "label = torch.argmax(logits)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb20d3a-cbba-4ab1-8584-d94e16589505",
   "metadata": {},
   "source": [
    "- We can apply this concept to calculate the so-called classification accuracy, which computes the percentage of correct predictions in a given dataset\n",
    "- To calculate the classification accuracy, we can apply the preceding `argmax`-based prediction code to all examples in a dataset and calculate the fraction of correct predictions as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3ecf9572-aed0-4a21-9c3b-7f9f2aec5f23",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_accuracy_loader(data_loader, model, device, num_batches=None):\n",
    "    model.eval()\n",
    "    correct_predictions, num_examples = 0, 0\n",
    "\n",
    "    if num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "\n",
    "            with torch.no_grad():\n",
    "                logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "            predicted_labels = torch.argmax(logits, dim=-1)\n",
    "\n",
    "            num_examples += predicted_labels.shape[0]\n",
    "            correct_predictions += (predicted_labels == target_batch).sum().item()\n",
    "        else:\n",
    "            break\n",
    "    return correct_predictions / num_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7165fe46-a284-410b-957f-7524877d1a1a",
   "metadata": {},
   "source": [
    "- Let's apply the function to calculate the classification accuracies for the different datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "390e5255-8427-488c-adef-e1c10ab4fb26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: mps\n",
      "Training accuracy: 46.25%\n",
      "Validation accuracy: 45.00%\n",
      "Test accuracy: 48.75%\n"
     ]
    }
   ],
   "source": [
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")\n",
    "elif torch.backends.mps.is_available():\n",
    "    # Use PyTorch 2.9 or newer for stable mps results\n",
    "    major, minor = map(int, torch.__version__.split(\".\")[:2])\n",
    "    if (major, minor) >= (2, 9):\n",
    "        device = torch.device(\"mps\")\n",
    "    else:\n",
    "        device = torch.device(\"cpu\")\n",
    "else:\n",
    "    device = torch.device(\"cpu\")\n",
    "\n",
    "print(\"Device:\", device)\n",
    "\n",
    "model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n",
    "\n",
    "torch.manual_seed(123) # For reproducibility due to the shuffling in the training data loader\n",
    "\n",
    "train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30345e2a-afed-4d22-9486-f4010f90a871",
   "metadata": {},
   "source": [
    "- As we can see, the prediction accuracies are not very good, since we haven't finetuned the model, yet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4a9d15-8fc7-48a2-8734-d92a2f265328",
   "metadata": {},
   "source": [
    "- Before we can start finetuning (/training), we first have to define the loss function we want to optimize during training\n",
    "- The goal is to maximize the spam classification accuracy of the model; however, classification accuracy is not a differentiable function\n",
    "- Hence, instead, we minimize the cross-entropy loss as a proxy for maximizing the classification accuracy (you can learn more about this topic in lecture 8 of my freely available [Introduction to Deep Learning](https://sebastianraschka.com/blog/2021/dl-course.html#l08-multinomial-logistic-regression--softmax-regression) class)\n",
    "\n",
    "- The `calc_loss_batch` function is the same here as in chapter 5, except that we are only interested in optimizing the last token `model(input_batch)[:, -1, :]` instead of all tokens `model(input_batch)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "2f1e9547-806c-41a9-8aba-3b2822baabe4",
   "metadata": {
    "id": "2f1e9547-806c-41a9-8aba-3b2822baabe4"
   },
   "outputs": [],
   "source": [
    "def calc_loss_batch(input_batch, target_batch, model, device):\n",
    "    input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "    logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "    loss = torch.nn.functional.cross_entropy(logits, target_batch)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a013aab9-f854-4866-ad55-5b8350adb50a",
   "metadata": {},
   "source": [
    "The `calc_loss_loader` is exactly the same as in chapter 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b7b83e10-5720-45e7-ac5e-369417ca846b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as in chapter 5\n",
    "def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
    "    total_loss = 0.\n",
    "    if len(data_loader) == 0:\n",
    "        return float(\"nan\")\n",
    "    elif num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        # Reduce the number of batches to match the total number of batches in the data loader\n",
    "        # if num_batches exceeds the number of batches in the data loader\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            total_loss += loss.item()\n",
    "        else:\n",
    "            break\n",
    "    return total_loss / num_batches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56826ecd-6e74-40e6-b772-d3541e585067",
   "metadata": {},
   "source": [
    "- Using the `calc_closs_loader`, we compute the initial training, validation, and test set losses before we start training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
    "outputId": "49df8648-9e38-4314-854d-9faacd1b2e89"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 2.453\n",
      "Validation loss: 2.583\n",
      "Test loss: 2.322\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n",
    "    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
    "    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
    "    test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)\n",
    "\n",
    "print(f\"Training loss: {train_loss:.3f}\")\n",
    "print(f\"Validation loss: {val_loss:.3f}\")\n",
    "print(f\"Test loss: {test_loss:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e04b980b-e583-4f62-84a0-4edafaf99d5d",
   "metadata": {},
   "source": [
    "- In the next section, we train the model to improve the loss values and consequently the classification accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "456ae0fd-6261-42b4-ab6a-d24289953083",
   "metadata": {
    "id": "456ae0fd-6261-42b4-ab6a-d24289953083"
   },
   "source": [
    "## 6.7 Finetuning the model on supervised data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a9b099b-0829-4f72-8a2b-4363e3497026",
   "metadata": {},
   "source": [
    "- In this section, we define and use the training function to improve the classification accuracy of the model\n",
    "- The `train_classifier_simple` function below is practically the same as the `train_model_simple` function we used for pretraining the model in chapter 5\n",
    "- The only two differences are that we now \n",
    "  1. track the number of training examples seen (`examples_seen`) instead of the number of tokens seen\n",
    "  2. calculate the accuracy after each epoch instead of printing a sample text after each epoch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979b6222-1dc2-4530-9d01-b6b04fe3de12",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/15.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "Csbr60to50FL",
   "metadata": {
    "id": "Csbr60to50FL"
   },
   "outputs": [],
   "source": [
    "# Overall the same as `train_model_simple` in chapter 5\n",
    "def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
    "                            eval_freq, eval_iter):\n",
    "    # Initialize lists to track losses and examples seen\n",
    "    train_losses, val_losses, train_accs, val_accs = [], [], [], []\n",
    "    examples_seen, global_step = 0, -1\n",
    "\n",
    "    # Main training loop\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()  # Set model to training mode\n",
    "\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward() # Calculate loss gradients\n",
    "            optimizer.step() # Update model weights using loss gradients\n",
    "            examples_seen += input_batch.shape[0] # New: track examples instead of tokens\n",
    "            global_step += 1\n",
    "\n",
    "            # Optional evaluation step\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader, device, eval_iter)\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Calculate accuracy after each epoch\n",
    "        train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "        print(f\"Training accuracy: {train_accuracy*100:.2f}% | \", end=\"\")\n",
    "        print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "        train_accs.append(train_accuracy)\n",
    "        val_accs.append(val_accuracy)\n",
    "\n",
    "    return train_losses, val_losses, train_accs, val_accs, examples_seen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9624cb30-3e3a-45be-b006-c00475b58ae8",
   "metadata": {},
   "source": [
    "- The `evaluate_model` function used in the `train_classifier_simple` is the same as the one we used in chapter 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bcc7bc04-6aa6-4516-a147-460e2f466eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as chapter 5\n",
    "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "    model.train()\n",
    "    return train_loss, val_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e807bfe9-364d-46b2-9e25-3b000c3ef6f9",
   "metadata": {},
   "source": [
    "- The training takes about 5 minutes on a M3 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "X7kU3aAj7vTJ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "X7kU3aAj7vTJ",
    "outputId": "504a033e-2bf8-41b5-a037-468309845513"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Step 000000): Train loss 2.153, Val loss 2.392\n",
      "Ep 1 (Step 000050): Train loss 0.617, Val loss 0.637\n",
      "Ep 1 (Step 000100): Train loss 0.523, Val loss 0.557\n",
      "Training accuracy: 70.00% | Validation accuracy: 72.50%\n",
      "Ep 2 (Step 000150): Train loss 0.561, Val loss 0.489\n",
      "Ep 2 (Step 000200): Train loss 0.419, Val loss 0.397\n",
      "Ep 2 (Step 000250): Train loss 0.409, Val loss 0.353\n",
      "Training accuracy: 82.50% | Validation accuracy: 85.00%\n",
      "Ep 3 (Step 000300): Train loss 0.333, Val loss 0.320\n",
      "Ep 3 (Step 000350): Train loss 0.340, Val loss 0.306\n",
      "Training accuracy: 90.00% | Validation accuracy: 90.00%\n",
      "Ep 4 (Step 000400): Train loss 0.136, Val loss 0.200\n",
      "Ep 4 (Step 000450): Train loss 0.153, Val loss 0.132\n",
      "Ep 4 (Step 000500): Train loss 0.222, Val loss 0.137\n",
      "Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
      "Ep 5 (Step 000550): Train loss 0.207, Val loss 0.143\n",
      "Ep 5 (Step 000600): Train loss 0.083, Val loss 0.074\n",
      "Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
      "Training completed in 1.07 minutes.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n",
    "\n",
    "num_epochs = 5\n",
    "train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n",
    "    model, train_loader, val_loader, optimizer, device,\n",
    "    num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n",
    ")\n",
    "\n",
    "end_time = time.time()\n",
    "execution_time_minutes = (end_time - start_time) / 60\n",
    "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1261bf90-3ce7-4591-895a-044a05538f30",
   "metadata": {},
   "source": [
    "- Similar to chapter 5, we use matplotlib to plot the loss function for the training and validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "cURgnDqdCeka",
   "metadata": {
    "id": "cURgnDqdCeka"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_values(epochs_seen, examples_seen, train_values, val_values, label=\"loss\"):\n",
    "    fig, ax1 = plt.subplots(figsize=(5, 3))\n",
    "\n",
    "    # Plot training and validation loss against epochs\n",
    "    ax1.plot(epochs_seen, train_values, label=f\"Training {label}\")\n",
    "    ax1.plot(epochs_seen, val_values, linestyle=\"-.\", label=f\"Validation {label}\")\n",
    "    ax1.set_xlabel(\"Epochs\")\n",
    "    ax1.set_ylabel(label.capitalize())\n",
    "    ax1.legend()\n",
    "\n",
    "    # Create a second x-axis for examples seen\n",
    "    ax2 = ax1.twiny()  # Create a second x-axis that shares the same y-axis\n",
    "    ax2.plot(examples_seen, train_values, alpha=0)  # Invisible plot for aligning ticks\n",
    "    ax2.set_xlabel(\"Examples seen\")\n",
    "\n",
    "    fig.tight_layout()  # Adjust layout to make room\n",
    "    plt.savefig(f\"{label}-plot.pdf\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "OIqRt466DiGk",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 307
    },
    "id": "OIqRt466DiGk",
    "outputId": "b16987cf-0001-4652-ddaf-02f7cffc34db"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAToJJREFUeJzt3Qd4U/X6B/Bv0z1pS3cpFGgpe+8hCMhQUdwXvYJcxxXRi6LXKw4Q+StuUEEQvYobEAW8Ciiy95BhGWVTWqALSume5/+8vzRpUlroTtJ+P89znuScnCQnp2ne85uvnaZpGoiIiMgq6Sx9AERERFQ+BmoiIiIrxkBNRERkxRioiYiIrBgDNRERkRVjoCYiIrJiDNRERERWjIGaiIjIijFQExERWTEGaiKqkEGDBuHpp5/m2SKqYwzURHXkoYcegp2d3VXLiBEj+DcgonI5lP8QEdU0CcpffPGF2TZnZ2eeaCIqF0vURHVIgnJQUJDZ4uPjox7bsGEDnJycsHnzZuP+b7/9NgICApCYmKjWV69ejf79+8Pb2xuNGzfGrbfeipMnTxr3P3PmjCqlL1myBAMGDICrqyt69OiBY8eOYffu3ejevTs8PDwwcuRIJCcnm5X2R48ejenTp8Pf3x9eXl54/PHHkZeXV+5nyc3NxXPPPYfQ0FC4u7ujV69e6jMYxMbGYtSoUerzyePt2rXDypUry329jz/+GJGRkXBxcUFgYCDuvvtu42NFRUWYOXMmmjdvrj5Tp06dsHTpUrPnHzx4UH0u+Xzy/AcffBApKSlmVff/+te/8Pzzz8PX11ed+1dffbVCfzciS2KgJrKyNmAJMGlpadi3bx9eeeUVfPbZZyrwiMzMTEyePBl79uzB2rVrodPpcMcdd6hAZmratGl4+eWXsXfvXjg4OOD+++9XAeqDDz5QFwInTpzA1KlTzZ4jr3fkyBEVbL///nv89NNPKnCX58knn8T27duxaNEi/PXXX7jnnntUjcHx48fV4xMnTlTBfNOmTYiOjsZbb72lgmhZ5PNIEH3ttddw9OhRdUFyww03GB+XIP3VV19h/vz5OHToEJ555hn8/e9/x8aNG9Xjly9fxuDBg9GlSxf1WvJ8ubi59957zd7nyy+/VBcNO3fuVBdB8n5r1qyp9N+KqE5Jmksiqn3jxo3T7O3tNXd3d7Pl9ddfN+6Tm5urde7cWbv33nu1tm3bao8++ug1XzM5OVnS1GrR0dFq/fTp02r9s88+M+7z/fffq21r1641bps5c6YWFRVldmy+vr5aZmamcdu8efM0Dw8PrbCwUK0PHDhQmzRpkrofGxurPsu5c+fMjmfIkCHalClT1P0OHTpor776aoXOzY8//qh5eXlpV65cueqxnJwczc3NTdu2bZvZ9ocfflgbM2aMuj9jxgxt2LBhZo/HxcWpz3306FHj8ffv399snx49emj/+c9/KnSMRJbCNmqiOnTjjTdi3rx5ZtukGtZAqr6//fZbdOzYEc2aNcOsWbPM9pXSqpSEpUQo1bqGkvTZs2fRvn17437yfANDabxDhw5m25KSksxeW6qT3dzcjOt9+vRBRkYG4uLi1LGYkhJyYWEhWrVqZbZdStBSJS+khDxhwgT8/vvvGDp0KO666y6z4zJ10003qfdo0aKFKpXLIjUFcjxS+s/KylL7mJJqeSlBiwMHDmD9+vVlltilacBwnKXfPzg4+KrzQGRtGKiJ6pBUu0ZERFxzn23btqnbS5cuqUWeYyBtvhLQPv30U4SEhKhALQG6dFuyo6Oj8b60WZe1rXR1eWVIALe3t8eff/6pbk0ZguUjjzyC4cOH49dff1XBWqqv33vvPTz11FNXvZ6np6eqppdqd9lXLkak/Vja1eW9hLyOtIeX1RFP9pFzI9XrpUkwLuu81MR5IKoLDNREVkRKf9L+KoF48eLFGDduHP744w/VFn3x4kXVfiuPSUcxsWXLlhp7bymVZmdnq85aYseOHSrohoWFXbWvlGSlRC2lUcOxlEWeK53SZJkyZYo69rICtZC2dCl5yyJt7NJhbt26daokLQFZag0GDhxY5nO7du2KH3/8EeHh4ep1iOoTfqOJ6pBUDSckJJj/Ezo4wM/PTwU+6SAlpdDx48er6l+prpZS6L///W/Ve1qqlRcsWKBKiRK4XnjhhRo7NimVP/zww6oTmvQel2ApHcbkIqE0qUp+4IEHMHbsWHV8ErilF7l0SJPq5VtuuUV1jJNe2LJvamqqqppu06ZNme/9yy+/4NSpU6oDmXxO6R0uJd2oqChV2pbe5XIBI9uk17t0ttu6davqnS4XM9JxTS4CxowZY+zVLVXm0tFNOuOVLvUT2RIGaqI6JL2RTatihQSjmJgYvP7662pIkwQtIftJUJbgM2zYMNWGLIFH2n6lulue9+GHH6re4jVhyJAhaniUBEu5oJD3vdbwJRkP/n//93949tlnce7cOXWx0bt3bzVkTMiFhwTQ+Ph4FVDlwqN0m7uBlJ6ll7m8X05OjjoO6XkuQ7rEjBkz1LAxqT6XgC77Syn6xRdfVI9LM4AE7v/85z/qXMnxSxOBvGdZFxpEtsROepRZ+iCIyLJkHLUMcVq+fDn/FERWhpeaREREVoyBmoiIyIqx6puIiMiKsURNRERkxRioiYiIrBgDNRERkRVjoK6GuXPnqpmQJC2fpPjbtWsX6ivJgCRTNMp4VZl2sfQwHhnlJ9M+ythfmdlKZpcyZFEykOkwZZIMGVMr42Blcg3D9JAGkoVJZrqScyqzWkmGI1sg43slnaRMziFpKSVlpMwiZkrGB8u4Ypm0RGb8krmvDekrDWQSE5ksROa4lteRiU4KCgrM9pFpNmUMsczWJdORLly4ELZA5jiXyVDk7y+LzCW+atUq4+MN/fyU5c0331T/bzJ5jAHPE9R4ezkvpkvr1q3r7zmyWDoQG7do0SLNyclJ+/zzz7VDhw6pLEfe3t5aYmKiVh+tXLlSe+mll7SffvpJZSRatmyZ2eNvvvmm1qhRI2358uXagQMHtNtuu01r3ry5lp2dbdxnxIgRWqdOnbQdO3Zomzdv1iIiIozZj0RaWpoWGBioPfDAA9rBgwdV1idXV1ftk08+0azd8OHDtS+++EId9/79+7Wbb75Za9q0qZaRkWHc5/HHH9fCwsJUFqs9e/ZovXv31vr27Wt8vKCgQGvfvr02dOhQbd++feqc+/n5GbNRiVOnTqlMUpMnT9YOHz6sffTRRyqL1erVqzVr9/PPP2u//vqrduzYMZXR6sUXX9QcHR3VORMN/fyUtmvXLi08PFzr2LGjMWuZ4HnStGnTpmnt2rXTLly4YFwkk1x9PUcM1FXUs2dPbeLEicZ1SQUYEhKi0gfWd6UDdVFRkRYUFKS98847xm2XL1/WnJ2dVbAV8kWX5+3evdu4z6pVqzQ7OztjqsSPP/5Y8/HxUakeDSQFoWk6RluRlJSkPu/GjRuN50OC0g8//GDc58iRI2qf7du3q3X5sdDpdFpCQoJZqklJ/2g4J88//7z6gTJ13333qQsFWyR/b0nJyfNjLj09XYuMjNTWrFljll6U56kkUMtFf1nq4zli1XcV50SWrEFSvWsg0xTK+vbt29HQnD59Ws1fbXo+GjVqpJoDDOdDbqW6u3v37sZ9ZH85b5Ky0bCPTF8pqR4NZN5rqUKWuaJticxFbZrCUr4v+fn5ZudIquqaNm1qdo5kbm9DWkrD579y5QoOHTpk3Mf0NQz72Nr3TqYXlelQMzMzVRU4z485qbaVatnSf2uepxLStCZNcZIaVZrUpCq7vp4jBuoqkDzA8kNj+kcWsl464UJDYPjM1zofcivtQKWTUUggM92nrNcwfQ9bIIkjpE2xX79+xhzRcvxyASIXK9c6R9f7/OXtIz8wkvnK2kkea2kzlDY/yai1bNkytG3blufHhFzASMpP6fdQGr9HelIIkPZimTtf+j5IYUH6tqSnp9fLc8SkHES1UBo6ePBgjaagrC8kkcj+/ftVjcPSpUtV5quNGzda+rCsRlxcHCZNmoQ1a9aoDpVUNsnKZiAdFCVwSxKWJUuWGNO01icsUVeBZAmStHmlexHKelBQEBoaw2e+1vmQW8ldbEp6WEpPcNN9ynoN0/ewdpIWUrJfSUrHJk2aGLfL8UuTiSS+uNY5ut7nL28f6UVtCz9QUtKR3rPdunVTJUbJCPbBBx/w/BSTalv5P5GexlLjJItcyEiWNLkvJTp+j64mpWdJpyqpTevj/xoDdRV/bOSHRnLvmlZ3yrq0tzU0zZs3V19q0/Mh1UPS9mw4H3Ir/zjyQ2Swbt06dd7katiwjwwDk/YlAylZSClMchRbM+ljJ0FaqnLlc8k5MSXfF0dHR7NzJG3v0q5meo6katj0gkY+v/wwSPWwYR/T1zDsY6vfO/n7S0pKnp+SVKPyHZBaB8Mi/TqkDdZwn9+jq8kwz5MnT6rhofXyu1Tn3dfq0fAs6dW8cOFC1aP5scceU8OzTHsR1ifSC1WGMcgiX5v3339f3Y+NjTUOz5LPv2LFCu2vv/7Sbr/99jKHZ3Xp0kXbuXOntmXLFtWr1XR4lvTWlOFZDz74oBqyI+dYhkfYwvCsCRMmqOFpGzZsMBsykpWVZTZkRIZsrVu3Tg0Z6dOnj1pKDxkZNmyYGuIlw0D8/f3LHDLy73//W/VknTt3rs0MP3rhhRdUL/jTp0+r74isS6//33//XT3e0M9PeUx7fQueJ0179tln1f+afJe2bt2qhlnJ8CoZbVEfzxEDdTXIuDr5Msh4ahmuJeOD66v169erAF16GTdunHGI1iuvvKICrVzADBkyRI2VNXXx4kUVmD08PNQwiPHjx6sLAFMyBrt///7qNUJDQ9UFgC0o69zIImOrDeSi5YknnlBDkuQH4I477lDB3NSZM2e0kSNHqvHj8sMjP0j5+flX/S06d+6svnctWrQwew9r9o9//ENr1qyZOm75UZTviCFIi4Z+fioaqHmeNDVMKjg4WP2N5XdC1k+cOFFvzxGzZxEREVkxtlETERFZMQZqIiIiK8ZATUREZMUYqImIiKwYAzUREZEVY6AmIiKyYgzU1SAzKkkCc7klnid+l2oX/994jhrq98ii46hlrt+ffvoJMTExau7Uvn374q233lJTRpZHMqaMHz/ebJtk4snJyUFdk2kyJZ2jJBiQqeeI54nfJf6/WRJ/k+rnObJoiVomm5dMQzt27FBzqMocz8OGDVM5aq9FTu6FCxeMS2xsbJ0dMxERUYNJcym5REuXliVnsSRuuOGGG8p9np2dnc1kUyIiIqo3+ailKkL4+vpeN1OK5B6VzDuSDu6NN95Au3btKvQeklpx3759Kl2cTle9CgVJUi7OnTunqlOI54nfpdrD/zeeo/r0PZL4JWkzu3TpolKYXovVzPUtB33bbbepVIhbtmwpd7/t27fj+PHjKlm4BPZ3331XpUY8dOiQWf5fA+kwYNppQErrgwcPrrXPQUREVFG7du1Cjx49bCNQT5gwAatWrVJBuqyAWx5p127Tpg3GjBmDGTNmXPW49O6bPn16mSdHcpcSERHVNelf1bNnT9XHqmnTptYfqJ988kmsWLFClYybN29e6effc889qurg+++/v26JWqo7JDF4XFxcpS4IiIiIakp8fDzCwsIqFIss2utbrhEkSC9btgzr1q2rUpAuLCxEdHR0uaVjGbolvcQNi6enZw0cORERUQPoTCZDs7777jtVmpYAmpCQoLbLGDcZVy3Gjh2L0NBQNeZavPbaa+jduzciIiJUe/Y777yjqg4eeeQRS34UIiKi+heo582bp24HDRpktv2LL77AQw89pO6fPXvWrHd2amoqHn30URXUfXx80K1bN2zbtk1VZxMREdU3VtFGba3tAkTU8EhzmnRSJaoOR0dH2Nvb10gssqpx1EREliJlFqmpkyY1oprg7e2tJueSSbqqg4G6OrIvA2d3AI2aAEHtq/VSRGRZhiAtsyO6ublV+8eVGvZFX1ZWFpKSktR6dYcCM1BXx7r/A3Z/CvR6HBj5VrVeiogsW91tCNKNGzfmn4KqzdAhWoK1fK+uVQ1+PUxzWR3h/fS3Z7ZW62WIyLIMbdJSkiaqKYbvU3X7PDBQV0ez4kCdeBDIulStlyIiy2N1N1nj94mBujo8AgC/VtIiAZzdXiN/ECIiIlMM1NUV3l9/y+pvIqonwsPDMXv27Arvv2HDBlV6rO0e8wsXLlQ9qRsaBuqaqv4+s7n6fw0iokqQ4HitRZISVcXu3bvx2GOPVXj/vn37qiQTMqsk1Tz2+q6pEnVCtH64lmvDu9ojIsuQ4GiwePFiTJ06FUePHjVu8/DwMBsyJL3br5f7WPj7+1fqOJycnNR4YaodLFFXl2cQ0DiiuJ16R438UYiIKkKCo2GR0qyUog3rMTExKoeCpA+WqZYlQZGkET558iRuv/12BAYGqkAuuZD/+OOPa1Z9y+t+9tlnuOOOO1RP5sjISPz888/lVn0bqqh/++03lYZY3mfEiBFmFxYFBQX417/+pfaTIXH/+c9/MG7cOIwePbrSU1G3bNlSXSxERUXh66+/Nrs4kVoFSSMpnz8kJES9p8HHH3+sPouLi4s6H3fffbdVfvEYqGsCq7+J6uekFXkFFllqcmbnF154AW+++SaOHDmCjh07IiMjAzfffDPWrl2Lffv2qQA6atQolVfhWqZPn457770Xf/31l3r+Aw88gEuXyh/tIhN+vPvuuypwSgpjef3nnnvO+Phbb72Fb7/9VuV22Lp1K65cuYLly5dX6rMtW7YMkyZNwrPPPouDBw/in//8J8aPH4/169erx3/88UfMmjULn3zyCY4fP65ev0OHDuqxPXv2qKAtiZ6kFmL16tW44YYbYI1Y9V1T1d97vwRiOZ6aqL7Izi9E26m/WeS9D782HG5ONfPzLIHopptuMq77+vqiU6dOxvUZM2aogCclZEk7XB5JlDRmzBh1/4033sCHH36IXbt2qUBfFhk7PH/+fFXaFfLaciwGH330EaZMmaJK6WLOnDlYuXJlpT7bu+++q47riSeeUOuTJ0/Gjh071PYbb7xRXRxI7cLQoUPV3NtSsu7Zs6faVx5zd3fHrbfeqmoemjVrhi5dusAasURdkyXqCweAnLQaeUkioprQvXt3s3UpUUvJVqqkpdpZqqWltH29ErWUxg0kwHl5eRmnyCyLVJEbgrRhGk3D/mlpaUhMTDQGTSEzd0kVfWUcOXIE/foV//4Wk3XZLu655x5kZ2ejRYsWKuuiXJBIlbuQixcJzvLYgw8+qEr3UgtgjViirgmNQgGf5kDqaeDsTqDVsBp5WSKyHFdHe1WytdR71xQJqqYkSK9Zs0aVOiMiItRUl9I2m5eXd83XkRKpKWmTLioqqtT+dZ2sMSwsTFVrSxu8fGYpeb/zzjvYuHGjKkXv3btXta///vvvqiOetGdLj3drGwLGEnVNiboZaDUCcDL/pyAi2ySBRaqfLbHU5gxp0h4s1cVS5SzttVI1fObMGdQl6fgmnbckKBpIj3QJnJXRpk0b9XlMyXrbtm2N63IhIm3wUlUvQXn79u2Ijo5Wj0kPeKkWf/vtt1Xbu5yHdevWwdqwRF1TRrxRYy9FRFRbpJfzTz/9pIKXXBC88sor1ywZ15annnoKM2fOVKX61q1bqzbr1NTUSl2k/Pvf/1Yd3KRtWQLu//73P/XZDL3Ypfe5XAD06tVLVcV/8803KnBLlfcvv/yCU6dOqQ5kPj4+qn1czoP0HLc2DNRERA3I+++/j3/84x9qkhI/Pz81LEp6XNc1eV9JLTp27FjVPi0TrAwfPrxSWaZGjx6NDz74QFXjS+/v5s2bq17kgwYNUo9LFbb0eJdOZhKwpQZBgrkMB5PHJKhLdXdOTo66gPn+++/Rrl07WBs7ra4bDSwsPj5etVvExcWhSZMm1X69gsIi2Ov0swApl+MAnQPgVb38o0RUd+SH+vTp0+qHXsbUUt2T0qxUZUsJWXqi1/fvVXwlYhHbqKvh+aUH0HXGGhw8V3w1uvpFYHZ7YNeC6rwsEVG9Fxsbi08//RTHjh1TbcYTJkxQQe3++++39KFZHQbqakjNyseVnAJsPFY8RCGwHWBnD2RdrKE/DxFR/aTT6VQbssyMJkOqJFhL27KUqskc26irYWArf6w5nIiNx5Lx5OBIoN1ooO1tgLNndV6WiKjek2rf0j22qWwM1NUM1GLv2ctIy85HI1cOzSIioprFqu9qCPN1Q0t/dxQWadh6IsX8QQsMdyAiovqHgbqaBrYKULcbjybrN5z7E/h0MPDVbdX+4xARETFQV9PAKH31t7RTq5FuLt76YB23E8jP5jeMiIiqhYG6mno194Wzgw4JV3JwNDEd8G0BeAYDhXlAfMn0eERERDYXqGX6OOmaL5OjBwQEqFlmZAL16/nhhx/UlHMygFxmmqlsarSa5OJojz4tG5dUf8vEJ5L2Upxhj0YiIrLhQC0ZTCZOnKjyh0pmE8lfOmzYMGRmZpb7nG3btqmcqA8//LBKei7BXRZJGm7p3t9S/W2W9vLMFosdExFRRcmUm08//bRxPTw8HLNnz77mc2Q2xuXLl1f7JNfU61yLTBPauXNn2CqLBurVq1erLC4yt6okMpfB75IT9c8//yz3OTKvqyQql8nYZWC8TDXXtWtXlXTc0oF695lLyMwtKClRS9V3fo7FjouI6jdJrCG/h2XZvHmzCoKSFaqyJKuVzL1dF8HywoULGDlyZI2+V31jVW3Ukkxc+Pr6lruPpCiTLCmmZCJ32V6W3NxcNeG8YUlPT6/howaa+7mjqa8b8gs1bDt5EWgcAXgEAoW5+o5lRES1QGoWpTZS5o0uTZJTdO/eHR07dqz06/r7+6tsU3VB0mw6OzvXyXvZKp01TcguVS8ylVz79u3L3U+yrUgeU1OyLtvLaweX3KeGxTRPaU2Rq9aS6u8kfTs1q7+JqJbdeuutKqhKbaSpjIwM1ZdHAvnFixdVc2FoaKgKvtKvR7JEXUvpqu/jx4+rdJDSL0h+Q+XioKxsWK1atVLv0aJFC5U+U5ozhRzf9OnTceDAAfV7KYvhmEtXfctUooMHD1bpKCXL1WOPPaY+j4HUwkpzp2TMCg4OVvtIE6rhvSoab1577TWVDEMuEqSkLzW8Bnl5eXjyySfV68tnlrSYEkuEjO6R2oGmTZuq54aEhOBf//oXGkSglhMt7cyLFi2q0dedMmWKKqkblsOHD6M2GAL1hqPFw7TCi9upY9lOTWTT8jIrvxQWlDxf7su20sM1y3tuJTg4OKg0kRL0TBMhSpCWtI4SoCWDU7du3fDrr7+q31gJfA8++CB27dpV4aB25513wsnJCTt37sT8+fNVUC5NOgXLcchvrDRRSsKNWbNmqcfuu+8+PPvss6qZU6q6ZZFtpUn/JKkhlfzQUv0un+OPP/5QQdPU+vXrcfLkSXX75ZdfqvctfbFyLXJ87733ngr20jQg73nbbbepCxLx4Ycf4ueff8aSJUtUB+dvv/1WXbyIH3/8UX2uTz75RO0vFxly8VPvpxCVP4Ik8d60adN1031JNUliYqLZNlmX7WWRKx7TapXayrsqPb+d7HWIT83GqZRMtGxW3E4dtxsoyAUcWLVDZJPeCKn8c+5ZCLS7Q38/5n/ADw8B8psw/teSfWZ3KDuBz6v6JsCKktzS77zzjuqca8jDLNXed911l7Em8bnnnjPu/9RTT+G3335TQahnz57XfX0JlDExMeo5UnoUb7zxxlXtyi+//LLxvgQ1eU8peD3//POqdOzh4aEuLMr7rRbfffedurD46quv4O6un5J5zpw5qi3+rbfeMtamSiCX7ZK7WkYA3XLLLVi7di0effTRCp0zCdBysfG3v/1NrctrS9CXWoS5c+eqvlKSn7p///6qxC8lagN5TD6DNME6OjqqknVFzqPNlqjlClCC9LJly7Bu3TqVs/N6+vTpo/4gpqQaRrZbkruzA3o09ykZpuUfBbj5AQXZwLm9Fj02Iqq/JFD17dsXn3/+uVo/ceKE6kgm1d5CStbS6VZKfdL/RwKmBF0JOBVx5MgRlUDDEKRFWb+3ixcvVk2XEsTkPSRwV/Q9TN9LOhYbgrTo16+fKtWbDt2VkrkEaQOpok5KKs5ieB1SWDt//rx6XVOyLu9vqF7fv38/oqKiVLX277//btzvnnvuQXZ2tqrelwsDiV8FBSY1KPWtRC3V3XIFtWLFClVtYmhnlitAuQITUq0jbSuG9oFJkyZh4MCBqtpCrqLkim3Pnj1YsMDyOaCl+nvriYtqmNY/+jfXV38fXqGv/m5m2QsJIqqiF89X/jn2JjVorUfpX8OuVLno6ega+5NIUJaSspQGpTTdsmVL9TsppLQtVb1SWpRgLUFQ+gNJO2xNkc68DzzwgGqHlmpk+Q2X32b5na4Njo6OZutS6pVgXlNkJJHkxl61apWqUbj33ntVCXrp0qXqokUuGmS7FBKfeOIJY41G6eOqFyXqefPmqXZjqa6RKyLDIldmBnJFJu0ZBnLlKMFdArNcecmJkzaCa3VAqyuDovTzfu84dRE5+YX6qi5D9TcR2SYn98ov9iZlILkv2xxdK/a6VSCBRPI7y2+jVBtLdbgELyGpJG+//Xb8/e9/V7+ZUhI8duxYhV9bhsHGxcWZ/Q7L3Bel57eQ6uGXXnpJ9TSXauPY2Fjzj+vkpEr313sv6XBmOpfG1q1b1WeT0m1N8PLyUrUDpVNsyrppZ2PZT9rRpa1dYpK0TV+6dEk9JgVJqY6XtuwNGzaoCxXpBFcvS9SmnR/KIyehNKl6kMXaRAZ4ILiRCy6k5ahgPajt7UBoNyC4k6UPjYjqMalqlqAinWelaleqbg0kaEqBRoKptO2+//77ql9PRUfASElSenOPGzdOlRzl9SUgm5L3kEKVlKJltknpuCZVwqak3VpKqVKlLH2RpBa19LAsKZVPmzZNvZf0rE5OTlY1BdL5rfRon+qQeTjkfaTmQXp8Sy2EHJd0GhNyjqTQ2KVLF3WRIJ3apErf29tbdVqTC45evXqpHu7ffPONCtym7dj1ttd3fWA+TCsZ8AwEmnQzv7omIqoFUv2dmpqqqp5N25OlrViqcmW71F5KwJHhTRUlgUqCrrTLSqepRx55BK+//rrZPtJj+plnnlF9jiTwyUWBDM8yJZ3bZHKWG2+8UQ0pK2uImAQ+aT+XkqsE/LvvvhtDhgyp8QmtpN158uTJqie6NAfI0Czp5S0XHEIuIt5++21VOyDHcebMGTVVtZwLCdZSypY2bRmjLlXg//vf/9Qwsdpip1WkWFuPyMQA0sYgVTnX62FeFauiL2DCt3vRwt8d657V98AkIusmPY2ltCcdWmXcLFFtf68qE4tY1Kth/SL9YK+zw6nkTMRdykJYYTyw/SPAzh4Yde25c4mIiEpj1XcN83JxRLem+mFaG6T6W6YR3fsVEP2D+SQIREREFcBAXQsGRvmXjKcOaAf0nwzcLWMcG1QrAxER1QAG6lpg6FC27WQKcos0YOg0oNVwwL52xtgREVH9xUBdC9oGe8HPwxlZeYX480xqbbwFERE1EAzUtXFSdXa4oZVfyTCtokLgxFpg3ev6+0RklWpydiuiohr6PrHXdy3OUvbT3nMqUE8Z0Qr4YTyQmwa0vhkI6VJbb0tEVSCzZskYWZkDWsb4yrphZi+iypJRzzJFq0zYIt8r+T5VBwN1LRkQ4afSUsckpONCeh6CZa7vY6uBM1sZqImsjPyYylhXmSZTgjVRTZAJXCS7lny/qoOBupb4uDuhUxNv7I+7jE3HknFfs37FgXoL0Nc8tyoRWZ6UeuRHVTIhXW9OaqLrkexektazJmpmGKhrufe3BGqp/r5vUHFKtbPb9O3UupIUbURkHeRHVTIg1VYWJKKqYGeyWjSoeDz15uMpKAjoADh5AjlpQOKh2nxbIiKqRxioa1HHJt7wdnNEek4B9p3LAJr21j8g1d9EREQVwEBdi2TO7wGRJrOUhRdXf8ea50ElIiIqDwN1LRtUPEvZhmNJQLP+JYGa4zWJiKgCGKhr2YDiiU8OnruCZM82gKM7kJ0KJB2u7bcmIqJ6gIG6lgV4uqBdiJe6v/nUZaBpL/0DrP4mIqIKYKCuw97fajpRGU8t2KGMiIgqgIG6DgxsFaBuZeKTQtN2ao1pL4mI6No44Ukd6NLUG57ODkjNysdBrQU6RQ7XV4EX5AKOLnVxCEREZKMYqOuAo70O/SP9sOpgAjacSEOnB5bUxdsSEVE9wKrvOpxO1DhMi4iIqIIYqOvIDcWB+kDcZaRm5gHpicCh5WynJiKia2KgriMh3q5oFeiBIg3Yeuw88EFH4IdxwMUTdXUIRERkgywaqDdt2oRRo0YhJCREZa1Zvnz5NfffsGGD2q/0kpCQAFswKErf+1vaqRHWCwjqCGRdsvRhERGRFbNooM7MzESnTp0wd+7cSj3v6NGjKsG7YQkI0AdAW2mnlvHURQ/8CDy+uWQCFCIiImvr9T1y5Ei1VJYEZm9vb9ia7uE+cHOyR3J6Lo4kZaFdSCNLHxIREVk5m2yj7ty5M4KDg3HTTTdh61bbyUTl7GCPvi0bl8xSJvKzgbwsyx4YERFZLZsK1BKc58+fjx9//FEtYWFhGDRoEPbu3Vvuc3Jzc3HlyhXjkp6eDqsYpiVpL1c+D7zZFIj+waLHRERE1sumJjyJiopSi0Hfvn1x8uRJzJo1C19//XWZz5k5cyamT58O65pO9BD2xqYit7kHnAvz9NOJdhtn6UMjIiIrZFMl6rL07NkTJ06UP8RpypQpSEtLMy6HD1s2vWTTxm5o4eeOgiINB+w7lCTo4LzfRERUHwP1/v37VZV4eZydneHl5WVcPD09YS2Tn/yS2gTQOQJXzgGpZyx9WEREZIUsGqgzMjJUoJVFnD59Wt0/e/assTQ8duxY4/6zZ8/GihUrVAn64MGDePrpp7Fu3TpMnDgRtmRgcdrLP45fgRbaVb+RaS+JiMja2qj37NmDG2+80bg+efJkdTtu3DgsXLhQjZE2BG2Rl5eHZ599FufOnYObmxs6duyIP/74w+w1bEGfFo3h7KDD+bQcpLbrCd+4nfp26q4PWvrQiIjIythpWsNqHI2Pj1e9xePi4tCkSROLHcfYz3ep/NTzel/GyP1PAI2aAs9EW+x4iIjIOmORzbdR2yrDMK2lSaGAnT2QdhZIjbX0YRERkZVhoLZwoN4cm43CkC76jVL9TUREVN1ALUV1KbYb7Nq1S3XsWrBgQVVerkFq6e+OJj6uyCssQrxXcaA+w0BNREQ1EKjvv/9+rF+/Xt2XzFUylacE65deegmvvfZaVV6ywZGsX4ZS9abc4klczmy27EEREVH9CNQyNEomGhFLlixB+/btsW3bNnz77beqtzZVjCFQf5cQom+nvhwLpJXUVBAREVUpUOfn56uJRIQMj7rtttvU/datW6shVVQxfSP84GhvhyOXgFz/DoCDC5B8lKePiIiqF6jbtWunkmNs3rwZa9aswYgRI9T28+fPo3FjfXYouj4PZwd0b+ar7v8vaibwwlkgYghPHRERVS9Qv/XWW/jkk09U5qoxY8agU6dOavvPP/9srBKnys1S9utZB8BBX0tBRERUrZnJJECnpKSotJE+Pj7G7Y899piaMYwqcS6j/PHmqhhsP3UROfmFcHG01yfosLPjaSQioqqVqLOzs1WeZ0OQjo2NVfNwHz16FAEBksaRKioq0BOBXs7IyS/C+ZVvA3N7Awd/5AkkIqKqB+rbb78dX331lbp/+fJl9OrVC++99x5Gjx6NefPmVeUlGyzTYVqJ52OB5CNM0EFERNUL1Hv37sWAAQPU/aVLlyIwMFCVqiV4f/jhh1V5yQZtUJS+FuLzjN7AvV8Dg1+x9CEREZEtB+qsrCxjXufff/8dd955J3Q6HXr37q0CNlVOvwg/2OvssOaiP+KDhwLu7DlPRETVCNQRERFYvny5mkr0t99+w7Bhw9T2pKQkeHl5VeUlG7RGro7oEuat7m86lmLpwyEiIlsP1FOnTsVzzz2H8PBwNRyrT58+xtJ1ly7F81ZTpRjaqQ9H/wlseBPY+QnPIBERVS1Q33333Th79iz27NmjStQGQ4YMwaxZs3haq9FOnREXDWyYCez5nOeRiIiqNo5aBAUFqcWQRUsSX3Oyk6prF+KFxu5O2JgZCbgASI4BMlMAdz9+TYmIGrAqlaiLiopUlqxGjRqhWbNmavH29saMGTPUY1SFP4TODje08kcqvJDk2lK/kfmpiYgavCoFaklnOWfOHLz55pvYt2+fWt544w189NFHeOUVDi2qzixlYkdRG/2GM1sa/BeUiKihq1LV95dffonPPvvMmDVLdOzYEaGhoXjiiSfw+uuv1+QxNhj9I/zUzKGr0lviNicJ1FstfUhERGSLJepLly6plJalyTZ5jKqmsYczOoY2wq6i4nObdAjI4vkkImrIqhSoJVuWVH2XJtukZE1VNzAqABfRCBecmuk3xG7j6SQiasCqVPX99ttv45ZbbsEff/xhHEO9fft2NQHKypUra/oYG9x46g/XHsemvCjch1h9O3WbWy19WEREZEsl6oEDB+LYsWO44447VFIOWWQa0UOHDuHrr7+u+aNsQDo1aaRmKtucF6XfEMsOZUREDVmVx1GHhIRc1WnswIED+O9//4sFCxbUxLE1SA72OvSP9MPOv4p7ficcBLJTAdeSvN9ERNRwVKlETbVrUCt/JMMb8fZNAGhA7HaeciKiBsqigXrTpk0YNWqUKp1LXmZJ9HE9GzZsQNeuXeHs7KySgyxcuBD1dd7vTXmt9Bs48QkRUYNl0UCdmZmpepDPnTu3QvufPn1adWK78cYbsX//fjz99NN45JFHzOYbrw8CvFzQJtgL/yvsgyNRE4H2d1n6kIiIyBbaqKXD2LVIp7LKGDlypFoqav78+WjevDnee+89td6mTRts2bJFJQIZPnw46tssZfMutMMCXShmhXa29OEQEZEtlKhlbu9rLTLn99ixY2vtYGUI2NChQ822SYCW7fW2+vtYMoqKNEsfDhER2UKJ+osvvoAlJSQkIDAw0GybrF+5cgXZ2dlwdXW96jm5ublqMUhPT4ct6NbMBx7ODsjPvIS4bUvQzM8TaH2zpQ+LiIjqWL3v9T1z5kyzUn/btm1hCxztdegX0RiDdfvR7I/HgM3vWvqQiIjIAmwqUEv+68TERLNtsu7l5VVmaVpMmTIFaWlpxuXw4cOwFQNbBWBnURvE2YcBod0BjVXgREQNjU0FapmudO3atWbb1qxZY5zGtCwyjEsCuWHx9PSErRgY5Y8LaIyBWW8hbdDrUKm1iIioQbFooM7IyFDDrGQxDL+S+2fPnjWWhk07pz3++OM4deoUnn/+ecTExODjjz/GkiVL8Mwzz6A+CvV2RWSAB6Qv2ZYTKZY+HCIiamiBes+ePejSpYtaxOTJk9X9qVOnqvULFy4Yg7aQoVm//vqrKkXL+GsZpiV5sevb0Kyyen9viTkHJERb+nCIiKiO2Wlaw2r4jI+PR1hYmMr01aSJTNFp3TYfT8bT/12DrS6T4Kwrgt0LZwEnd0sfFhERVUNlYpFNtVE3RD3CfZHl6IsUzQt2RQVA3E5LHxIREdUhBmor5+Jojz4tG2NnUWv9BslPTUREDQYDtY20U+8oKh7/fWarpQ+HiIjqEAO1jQRqGU8ttHN/AnlZlj4kIiKqIwzUNiDczx06n3Bc0HxhV5QPxO+29CEREVEdYaC2EQOjArCjuFTNdmoiooaDgdqGZikzVn/HskMZEVFDwUBtI3q3aIy9dvoOZVr8n0B+jqUPiYiI6gADtY1wc3JAYHg7JGre0BXmAuf2WPqQiIioDjBQ21g7taH6m+3UREQNAwO1DRlk0k5deJrt1EREDQEDtQ1p6e+BU+5dkKfZIy23iPmpiYgaAAZqG2JnZ4fwqM7omPsZPgx5h/mpiYgaAAZqG2ynzoEzNh1LtvShEBFRHWCgtjH9IhrDQWeHUymZiEtIsfThEBFRLWOgtjGeLo4Y1MQOvzi9iKBPOwAFeZY+JCIiqkUM1Daoa5sIBNtdhGNhFpB40NKHQ0REtYiB2gYNigrEP/OewcCiecgN7GTpwyEiolrEQG2D2gR7ItajE2LzGmHPmVRLHw4REdUih9p8caq9YVqSo3rpn/HI3fAesCUaCGwPBMnSAfBvDTg48/QTEdUDDNQ2PEuZBGrXCzuBwj+BM5tLHtQ5AH6tSoK3uu0AeARY8pCJiKgKGKhtVP8IPzVMa1rWveio6462urPo5nwOkdoZuBVeAZIO65foJSVPcg/QB+6OfwM63WfJwyciogpioLZR3m5O+HBMFyzfF4CNcRFYmp4L5MsjGoJwCW11sejiFI+ebucRWXQGPjlxsMtMAk6uA5r2LXmh1Fhg8d+B0G7AqNkW/ERERFQWBmobdnOHYLVomobzaTnYf/Yy9p1Nxf44X2w95491OV2B4rTVrshBlF08+nteQNGZFgh0PIMuTb3RJu0vOCb8pQK8me+KS9zG6vMOgG9zQGdf9x+UiKgBY6CuJ53LQr1d1XJLx2C1Lb+wCDEX0rE/LhX74i6rIL4/xQX7r0QAVwAcOaT2C3TIwp2NX0Zzd3e4HjivgneopwPspORdmAccW13yRo5uQEBb83Zv6bjm6l3rn1EuRtJzC3AxIw8XM3KRkpGH7PwC9Aj3RRMft1p/fyIiS7HT5BewAYmPj0dYWBji4uLQpEkTNCSXs/KwX4J28bLv7GWkZav6cjOB7g64M/A8ertdQGuchl/mcdgnxwAF2WW/sEegvvPa0OlAk276bYX5+k5tdnblHk9uQSEuZUrgzUNKRq4+CGfqb1OK7xu3Z+Qhr7CozNfp1KQRRrQPxsj2QQj3c6/i2SEiss5YZBWBeu7cuXjnnXeQkJCATp064aOPPkLPnj3L3HfhwoUYP3682TZnZ2fk5BTX8V5HQw7Upcmf/szFrOLqcn3wPnz+CgqKzL8SEmtb+7thaGAGertfQGu7WPimH4OdzIqWft64X9Ej65Hm014FWPvdnyJs3zs4GnoXfmvyL1UKvpieC6e0kzic0xiJmYVIzymo9DF7ODugsYcTGrs7qcp6OWbTb3CbYC/c3D4IIzsEISLAs3oniIiollQmFlm86nvx4sWYPHky5s+fj169emH27NkYPnw4jh49ioCAsocTeXl5qcdNq36p8uS8NfdzV8udXfVflJz8Qhw6n6ZK24Yq83OXs3EkKQtHknT4CKEAQuHuNAAdmjSCp2c2XK+cgk/2Gfz48RlkFF1Qr/OawzaMdcjCppOX8eHR42qbP1Kx22Ui8jV7xGqBOOkYglMIRaJTU6S6NUeWVwt4ePmoINzYw1kFZD8VlJ3h5+mstrs4mreRJ6fn4vfDCVgVnYDtpy7iyIUranlvzTFEBnioUvbIDsFoHeTJ7wkR2SSLl6glOPfo0QNz5sxR60VFReoq46mnnsILL7xQZon66aefxuXLl6v0fixRV15SenFHteLA/Vf8ZWTmFZa7fyNXRwS626GdyyW4unvC3qepCrqtCo9j2K5H4CBzlJfHMwTwb6WvSjcsYb0AR5frHmdqZh7WHE7EyoMXsPVECvILS77a4Y3dVMCWwN0htBGDNhFZlM1Ufefl5cHNzQ1Lly7F6NGjjdvHjRunAvGKFSvKDNSPPPIIQkNDVVDv2rUr3njjDbRr167M98jNzVWLwblz59C2bVtWfVdDYZGG40npiI5Pg4O9nSrx6ku/zvBxc4KTwzVmpi0q0leXJx8FUo4DKcW3si7Dx8ry3PGSyVoO/gRcPgtE3gQElv03F9L2vvZIIlYdTMDGY8nIKyhp35ZOd4aSdpcwb+h0rJEhorplM1XfKSkpKCwsRGBgoNl2WY+JiSnzOVFRUfj888/RsWNHpKWl4d1330Xfvn1x6NChMj/szJkzMX369Fr7DA2Rvc4OrYO81FJpOh3QqIl+iRhi/lh2anHwPlYcyI8B6QmAu3/JPn8t1vdEd3IvCdSXTgP7vtGPBZfFM1CV6qU6X5aM3AKsj0nCqoMXsD4mWVXlf7bltFqCvFwwon2QWqQHuXw2IiJrYtES9fnz51XJeNu2bejTp49x+/PPP4+NGzdi586d132N/Px8tGnTBmPGjMGMGTOuepwl6npm16fA2R1Anyf0QVns/Rr4+cmSfRqFAaFdSwJ3cGfA2UM9lJ1XiI3HJGgnYO2RJBXEDaQ9fFi7INzcPhi9WvjC0Z45a4iogZeo/fz8YG9vj8TERLPtsh4UFFSh13B0dESXLl1w4sSJMh+XHuGyGFy5IoOIyWb1fFS/mGrcEujyd+DcXiDpCJAWp18OFzed2OkA/zYqeLuGdsMIWe7pgJwiO9WWvTI6AWsOJ6ghYd/tPKsWbzdHDGsbiJHtg9Evwu/a1flERLXIooHayckJ3bp1w9q1a41t1NLuLOtPPmlSQroGqTqPjo7GzTffXMtHS1arWV/9InLTgQsHgPg9wLk/9cH7SjyQdEi/7Ptav19IF7g8tgFD2gSqJe9yALYn2mP1oQT8dihRje9esideLZ4uDhjaRoJ2EG5o5X9Vz3Miotpk8eFZMjRLOo91795djZ2W4VmZmZnGsdJjx45V1ePS1ixee+019O7dGxEREarDmYy/jo2NVR3MiODsCYT31y8G0s6tgrZh2WveEa0wH05zOmOgkwcGPr4FM25vj12nL+G36HisPJyihoAt23dOLW5O9hjcOkCVtCUxipuzvUqOwiGCRFRvA/V9992H5ORkTJ06VU140rlzZ6xevdrYwezs2bPQSQekYqmpqXj00UfVvj4+PqpELm3c0pObqEyeQUDrW/SLoed5fmbJ49IZragQKMpXs6w56HToG+GHvvufx6ue+3EprD125TfHjwmB2JwejF/+uqAWU072Ojja28HRQW51JevqVqe2O5muyz4Oduq9DPfNHjPsa3y9q1/L39MZrQI94eniyD88UT1m8XHUdY3jqKlM+Tn6YV8yhttgdkfgcqzZbkU6RyS6RmB7bjPsym6CBM0HSZoPEjUfXIInNNR9W7YMN4sK8tQvgfrbFv7ucHZgFT2RtbKZcdSWwEBNFZZ1CTi/T19Vfm6Pvt07K6Xc3TWdA5L7z0BK67+rpCh2l8/C+8RPyPAIx/nQkWqbzFeeX1CE/CJNrcukLPmGbepxw/bi9YJS6/J4gf51zqVmI+FK2VPnSnW8zDjXKsgTrQM99bdBngjzceO4cSIrYDO9vomsmpuvfqy3Yby3XNNKb3Jp55agLWO9MxKA9EQgMxl2RQUI8A9AQEjx+PLMbcCBWUBIV7S96aGS153TA8jL0lfJmy6+wYCHYT1Y//7XmR5XEq0cS8zA0YQrOJqYjqMJ6YhJSFfzqB9PylDLryippnd1tEerQA9V6pZqcxkL3yrIA/4ezmxnJ7JSDNREFSVB07upfml3h/ljki0sIwlwMZkExjMQ6PKgfn8DCfaX4/SZyKQ3+rXoHEuC+IBngaiR+u2ZKcD5/YB3GLz9o9Czua9aSt5CUyVtCdrGJTFdBe3s/EIciE9TiylfdycVwFXgLq4+l0WSoBCRZfG/kKgm2DsCjSRhiQnDhCulPbVH3xNdLReAjET9rVovvi9V7NK5zTAmPN9kfnSZ8GXxA0CTnsAja0q2fzoYKMiFnZsvgl19Eezmi0FujYGmvkBrXxS6+OBCvgdOpDvh0GVHRCcX4VhSBs5czFTD0XacuqQWs4/g7aqqzA1V5xLEW/p7VHlcuVxEyBS0kqHN/LZIf1tY9nbDYtguQ+Q4/Ss1FAzURHVdKjdMoXotBXn6uc8NAV1mWjPQ2QMB7YDGEebPSTxcfs5wuZYA0KR4GaRexwG4dTZyOtyPE0kZOH98H/wP/RdH8oPxYdZwVSqX6VYbpR3B6aNOWKR5IA0e0Ons0ayxmwqWZQVRY9CV9ULz7aUyqFZLS393/HNgS4zuHMoJaaheY2cyovpAqtSl41v2JSArFci6WHz/Uqn7l/T3DSX0uz8H2t+lv3/4Z2DJg/psZQ//rtq/pdq8/eLecM/VJ0wpgh2uaG5I1TyQDRfkwhE5mpP+FvrbXM0Ry4r6Y3uRfqx6MC7iVvvtSNK8saKoZHx7D7sYONgVqv1z4YQCnWFxQYGdEwp1zqqXvb29Ts3BLh3k9Lc6nL+cjfTi6V+DG7ngkQEt8LceYXBnVT3ZCHYmI2qIJXXTUvf15Gfrg7ZLo5JtklL0xpeNmcq83ZzQq0VjwMsXuJIL5KZBBw3edplquZYbbhiJjPYDVXB1i9+MgOXfocCvDaY+9KoKtPb2dnBbMA26i/pc5WWShGdF0pnOBbBzBnSuQL9JQO8JSM/Jx6LtJxGz5Sf8kdYSM37JwUfrjmNcn3A81DccPu5OFT8XRFaOVd9EDZGj69Vt6gGt9UtpE3eWdJiTDGfGUnk2UJBTvOQWr+eq9aCIfkCAPhEKCpoAHe+Dg2cwGnuUzLsP3xb6anyT5xkXI01fnS9LzmXjYzLJy6ORGcDGt5Dr6Y3hjl/gzKVsfLD2OL7edBi394zEowNaIMTbtebPHVEdY6Amoop3mJPStiE3eEUFtQfuXHD19geWlF+NX5hXKoDLbbaaOc4oJw3wi4KzXyTW3nsjVh9MwMfrj+OTS+ORs9sJG3e1QVHTvug7ZBSat4jiX5lsFtuoici2SUlfLiIkxqedg92sq6cTTnYIhq55PzRuOxgI7wd4N7vuGHWi2sQ2aiJqOIqDtLCT6vznT6shbEnRa5F1YjPCco7Bv+ACcHypfpGA7hUKu2b99FnXJIGL9KBn4CYrxapvIqpfZEa31jcjoLU+9e3J+PNY//v/UHB6C3rYHUFHu1NwvHIOiF6iX8Qzh0qGzKlOdt6ASTIgIktioCaieq1lkxC0/Mc/cf7yWHy2+TQe2XUcbQpj0EsXg4FOR9HcNRsu7sEwdnNb9jgQvwu4bQ7Q5lY0dBm5BTiZlKHG2p9IzkDcpSzVm1/G0ZcsOjU9reG+6WOuJtvkvrPJfckGR9fHQE1EDYL0AJ86qi2eGhyBL7e3wRfbzmBWVj7ssorg/9Z6PNy/Oe7vGQbPhGh973bTXvEHfwIOfK+vKpcq8+DOgEP9GQImM8alZOQZg7EhMJ9MzsCFtLITv9QEGRfv4qCDq5O9yvamAr6TPVzUffPA71p839fdGf0iGqN9SKMGk2CGncmIqEHKzC3Aot1x+GzzKWMw8nRxwEO9QvFwyzR4t+wF2BeXZVZMBPZ9U/JkmdVNhpfJ2HOzJcJ8bLqVKSrSEJ+ajRPJ6fpAnJSpArPcT8vOL/d5fh7OiAhwR0SAB8Ibu6tt2XmFyCkoRE5+kZpDPie/ELkm92XJzi9CrvG+fl95Tk3kbGzs7oQbWvljUJQ/BkT6q/nqbQnTXNbQySGi+i+voAgr9p/D/I0ncTJZP5GLs4MO93YPw2M3tECYrxuQdAQ4uR6I3apfpMRdHsmA5hcJtL5FTc5iJNGpjjqs5RYU4nRKpj4QF5eS5fZUcgZyC2QmmavJoUkaVAnGMj2r3KrF3xON3Eo67NVE6V2OIbc4aJsF/OL7uaaBPb/kvmyXz7Xt5EVVJW967J2aeKugPbCVPzo28ValdWvGQF1DJ4eIGg4pba45koiPN5zEgbjLapv82I/qGIzHB7VUmcWKdwTSz+vTnKYcB1KOFS/H9WlPDbr/A7h1lv5+Xibwbit9KfwfvwFObvrtkoRFSuCOLlU65is5+Wbtx4b7Zy9llTuvupO9Di383VVylZbGYOyhtkkVs61cXP0Zm4oNx5Kw8WiySu1qysfN0VjaviHS33yiHSvBQF1DJ4eIGh4p8W0/dRHzNpzE5uMpxu2DWwdgwqCW6BFeklL0KjIJS8oJfeD2bQ407a3ffuEA8MkNgGQze/4U8gv1VcTOi+6D05l1yPcKQ7ZXS2R6tsAVj+ZIdQtHinMzpNl5IadAX9KU/dWSV6gCsQTkpPTccg/F09mhJBAXB2O5lRoCay9tVlZCWg42HkvChqPJ2HI8xTgPvKG03SG0EQa18sfAqAB0DrOO0jYDdQ2dHCJq2A6eS8O8jSexMvqCsV21ezMf3NoxWGUFyy0VRHNKBVRDtW1+Xj5888/DI/8Stua3Us8VvzpNQTtdbLnvL8lPTmohOFkUghNyq4Uguqg5kuGjHndGHqI8shHm54XGweHGgBzlmAhf5yLYaUWALFILoO4XAkWFJfdNH2vcUr8Iqdo/uQ6wdzbv+X50lT4Nq7yGWgpMlnLWpQNeu9ElQ99WPifhE7j7vyWvu+514Oz2a7xmfsm6vZN+3HvkTUCvf151zuQiaG9sKjYeS1aB+/CFK2aPN3J1xIBIPwyKClDV5P6eliltM1DX0MkhIhLSLrpg00n8+Oc55BWW3cZbFXZ2Gpo4ZqC1QwIidRfQUnce4do5hBXFw68wSSVBKW1b+EScaz9BBeVWmXvgvvhuILA9MGFryU4fdgUunazcwUhCloH/1t+Xnu/z++unbH3uWMk+/x0GxBXP/V5RPf8J3Py2/r6kbH0vCrCzB6aZ5D5f9AAQ80vlXrfzA8Doj0vSwn7QSX+h8bfvAJfiZoq8TCRl67DheIoK3JuPJeNKTklpW7QP9cKgVgEYGOWvcpw71NGQMc5MRkRUg5r7uWPmnR3x9NBW+HLbGRxPylDDhdQiw4mM90vGEJf9eMl2GU8sndbsyutglpelD7aG9u/itvC+fW4AosL0+5x2ARxczGZnU9z9gLwMwE6nD4pyKxO4mK3LrSx2+vumc7g7eQDhAwBXfcndSErHbn76/aXnu7yv3BrWjYvJepMeJc939gJGvKnfbqrPRKD9neW8hqP5NvlcqmmhRcnzL53S9xvITQecPUu2L/snAk5txL1+rXCvfxQKh0TiFEKx8aIvfj7rgL/OZ+LguStqmbP+BLxcHFQPcgnaUlUe4FW1vgM1jcOziIjIthXkAokHgYxkIGpEyfa5vYHkI2U/x94ZBb4tccGxGaJzA7Hhkg8O5ATitBaMPOgvfNoEe6kOaRK0uzbzqdEJWlj1XUMnh4iIbDyAX5RaiaNA8rHi2+Le+oVld8Tb0eQfmJlzF/46l4ZGWjoG6/bhqBaGs06R6B/pp9q1b+0UAg/n6s0XxqpvIiIiB2cgsK1+MSWd0i7HmgTvY0ByjKpS792rH1Z06I+LGbmI2fIT+u2Yr6rLB+e8g1UHE/DboQQMbxckPfnqDKcQJSKihkVnr2/jlsW0qly69ksPeJn5zMMZ/VqFAAkDEO7bEsu79MOGo0lIvJIDnzqeBc0qZkSfO3cuwsPD4eLigl69emHXrl3X3P+HH35A69at1f4dOnTAypUr6+xYiYionrIr7lhn0GIg8NAv0N32gRp/LZ0JpVNhXbN4oF68eDEmT56MadOmYe/evejUqROGDx+OpKSkMvfftm0bxowZg4cffhj79u3D6NGj1XLw4ME6P3YiIqJ63+tbStA9evTAnDlz1HpRUZHq7PXUU0/hhRdeuGr/++67D5mZmfjll5Ixd71790bnzp0xf/78674fO5MREZGlVSYWWbREnZeXhz///BNDhw4tOSCdTq1v3769zOfIdtP9hZTAy9ufiIjIllm0M1lKSgoKCwsRGBhotl3WY2JiynxOQkJCmfvL9rLk5uaqxSA93XzydiIiImtm8Tbq2jZz5kw0atTIuLRtW6qbPhERkRWzaKD28/ODvb09EhMTzbbLelBQUJnPke2V2X/KlClIS0szLocPH67BT0BERFSPq76dnJzQrVs3rF27VvXcNnQmk/Unn3yyzOf06dNHPf70008bt61Zs0ZtL4uzs7NaDC5f1ueZvXDhQg1/GiIioooxxCCJedelWdiiRYs0Z2dnbeHChdrhw4e1xx57TPP29tYSEhLU4w8++KD2wgsvGPffunWr5uDgoL377rvakSNHtGnTpmmOjo5adHR0hd5v165d0sudC88BvwP8DvA7wO+AZulzIDHpeiw+M5kMt0pOTsbUqVNVhzAZZrV69Wpjh7GzZ8+qnuAGffv2xXfffYeXX34ZL774IiIjI7F8+XK0b9++Qu/XpUsXNaGKvL7p61aFdEyTNm+pTvf0NMnYQjxfNYTfMZ6v2sTvl+XOl5SkpdlWYpLVj6O2ZVeuXFEd1KTt28urOP8p8XzxO2Yx/J/k+aqP36963+ubiIjIljFQExERWTEG6mqQ3uQyR7lpr3Li+apJ/I7xfNUmfr9s43yxjZqIiMiKsURNRERkxRioiYiIrBgDNRERkRVjoK6GuXPnIjw8HC4uLiqvtkykQmXbtGkTRo0ahZCQENjZ2alJaqj8RDKSo10mVAgICFDT6x49epSnqxzz5s1Dx44d1bhWWWQ64VWrVvF8VdCbb76p/idNp2Umc6+++qo6R6ZL69atUVcYqKto8eLFmDx5suoBuHfvXnTq1EnlxU5KSqrZv1A9kZmZqc6RXNzQtW3cuBETJ07Ejh071Dz2+fn5GDZsmDqHdLUmTZqoYCO57ffs2YPBgwfj9ttvx6FDh3i6rmP37t345JNP1IUOXVu7du3U/NyGZcuWLagzVZ+lu2Hr2bOnNnHiRON6YWGhFhISos2cOdOix2UL5Gu3bNkySx+GzUhKSlLnbOPGjZY+FJvh4+OjffbZZ5Y+DKuWnp6uRUZGamvWrNEGDhyoTZo0ydKHZLWmTZumderUyWLvzxJ1FeTl5amr96FDhxq3ybzhsr59+/aavI4iUtMVCl9fX56N6ygsLMSiRYtU7UN5GfVIT2ptbrnlFrPfMSrf8ePHVdNdixYt8MADD6g8FHXF4kk5bFFKSor6QTAkDjGQ9ZiYGIsdF9U/MnG/tB3269evwolnGqLo6GgVmHNycuDh4YFly5ap5AlUNrmYkSY7qfqm65M+SAsXLkRUVJSq9p4+fToGDBiAgwcP1klCJgZqIisv9ciPQZ22h9kg+QHdv3+/qn1YunQpxo0bp9r6GayvFhcXh0mTJqn+D9IRlq5v5MiRxvvSni+Bu1mzZliyZAkefvhh1DYG6irw8/ODvb29SlFmStaDgoJq6m9DDdyTTz6JX375RfWYlw5TVD4nJydERESo+926dVMlxQ8++EB1lCJz0mwnnV67du1q3CY1hPI9mzNnDnJzc9XvG5XP29sbrVq1wokTJ1AX2EZdxR8F+TFYu3atWRWlrLNdjKpL+ttJkJbq23Xr1qF58+Y8qZUk/48ScOhqQ4YMUU0FUgNhWLp3767aXeU+g/T1ZWRk4OTJkwgODkZdYIm6imRollSvyRe8Z8+emD17turAMn78+Jr9C9WjL7bp1efp06fVj4J0kGratKlFj80aq7u/++47rFixQrV/JSQkqO2SB9fV1dXSh2d1pkyZoqom5XuUnp6uzt2GDRvw22+/WfrQrJJ8p0r3d3B3d0fjxo3ZD6Iczz33nJoHQqq7z58/r4blygXNmDFjUBcYqKvovvvuQ3JyMqZOnap+SDt37ozVq1df1cGM9GR864033mh2oSPkYkc6aZD5BB5i0KBBZqfliy++wEMPPcRTVYpU444dO1Z18pGLGWlDlCB900038VxRjYiPj1dB+eLFi/D390f//v3VPAdyvy4wexYREZEVYxs1ERGRFWOgJiIismIM1ERERFaMgZqIiMiKMVATERFZMQZqIiIiK8ZATUREZMUYqImIiKwYAzUR1Ro7OzssX76cZ5ioGhioieopmW5UAmXpZcSIEZY+NCKqBM71TVSPSVCWOcJNOTs7W+x4iKjyWKImqsckKEuOdNPFx8dHPSala0kAIpmnJCtXixYtsHTpUrPnSzrEwYMHq8clu9Jjjz2mMqGZ+vzzz9GuXTv1XpL2T1J0mkpJScEdd9wBNzc3REZG4ueffzY+lpqaqtIrSnIDeQ95vPSFBVFDx0BN1IC98soruOuuu3DgwAEVMP/2t7/hyJEj6jFJ2zp8+HAV2Hfv3o0ffvgBf/zxh1kglkAvaTklgEtQlyAcERFh9h7Tp0/Hvffei7/++gs333yzep9Lly4Z3//w4cNYtWqVel95PT8/vzo+C0RWTiOiemncuHGavb295u7ubra8/vrr6nH593/88cfNntOrVy9twoQJ6v6CBQs0Hx8fLSMjw/j4r7/+qul0Oi0hIUGth4SEaC+99FK5xyDv8fLLLxvX5bVk26pVq9T6qFGjtPHjx9fwJyeqX9hGTVSPSQ5wQ35rA19fX+P9Pn36mD0m6/v371f3pYTbqVMnuLu7Gx/v168fioqKcPToUVV1fv78eQwZMuSaxyD5oQ3ktby8vFQOaTFhwgRVot+7dy+GDRuG0aNHo2/fvtX81ET1CwM1UT0mgbF0VXRNkTblinB0dDRblwAvwV5I+3hsbCxWrlyJNWvWqKAvVenvvvturRwzkS1iGzVRA7Zjx46r1tu0aaPuy620XUtbtcHWrVuh0+kQFRUFT09PhIeHY+3atdU6BulINm7cOHzzzTeYPXs2FixYUK3XI6pvWKImqsdyc3ORkJBgts3BwcHYYUs6iHXv3h39+/fHt99+i127duG///2vekw6fU2bNk0F0VdffRXJycl46qmn8OCDDyIwMFDtI9sff/xxBAQEqNJxenq6CuayX0VMnToV3bp1U73G5Vh/+eUX44UCEekxUBPVY6tXr1ZDpkxJaTgmJsbYI3vRokV44okn1H7ff/892rZtqx6T4VS//fYbJk2ahB49eqh1aU9+//33ja8lQTwnJwezZs3Cc889py4A7r777gofn5OTE6ZMmYIzZ86oqvQBAwao4yGiEnbSo8xknYgaCGkrXrZsmerARUTWi23UREREVoyBmoiIyIqxjZqogWKrF5FtYImaiIjIijFQExERWTEGaiIiIivGQE1ERGTFGKiJiIisGAM1ERGRFWOgJiIismIM1ERERFaMgZqIiAjW6/8BZX/3EzbaTIMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_losses))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_losses, val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd28174-1836-44ba-b6c0-7e0be774fadc",
   "metadata": {},
   "source": [
    "- Above, based on the downward slope, we see that the model learns well\n",
    "- Furthermore, the fact that the training and validation loss are very close indicates that the model does not tend to overfit the training data\n",
    "- Similarly, we can plot the accuracy below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "yz8BIsaF0TUo",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 307
    },
    "id": "yz8BIsaF0TUo",
    "outputId": "3a7ed967-1f2a-4c6d-f4a3-0cc8cc9d6c5f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAEiCAYAAADONmoUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAU6JJREFUeJztnQdYk9cXxl/ZIjhw40TFrbj33qNWrbtaqVr9a11ttbVaV22tdmmts9qqXe7dukq17r03bsGB4gSRTf7PuTEhICCBQCB5f4/fY+6XL993c0nyfufcc8/JotFoNCCEEEJIumOT/pckhBBCiEARJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCCCHETFCECSGEEDNBESaEJEiTJk3wwQcfcHQISUMowoSkEe+++y6yZMnyytamTRuOOSFEYaf9jxCSFojgLlmyJM4+R0dHDjYhREFLmJA0RAS3QIECcbZcuXKp53bt2gUHBwfs3btXf/w333yDfPny4f79+6q9bds2NGjQADlz5kTu3Lnxxhtv4Nq1a/rjb968qazrVatWoWHDhsiaNStq1qyJy5cv4+jRo6hRowZcXFzQtm1bBAYGxrHSO3XqhM8//xx58+ZF9uzZMXjwYERERCT6XsLDwzF69GgUKlQI2bJlQ+3atdV70HHr1i106NBBvT95vkKFCtiyZUui55s3bx48PT3h5OSE/Pnzo2vXrvrnYmJiMG3aNHh4eKj35OXlhTVr1sR5/blz59T7kvcnr3/nnXfw8OHDOO70ESNG4JNPPoGbm5sa+8mTJyfr70ZIekERJsTMc64iHs+ePcPJkycxYcIE/Pzzz0pUhJCQEHz00Uc4duwYduzYARsbG3Tu3FmJlCGTJk3C+PHjceLECdjZ2eHtt99W4jNr1iwl8levXsXEiRPjvEbOd/HiRSWky5cvx7p165QoJ8awYcNw8OBBrFixAmfOnEG3bt2UpX/lyhX1/NChQ5VQ79mzB2fPnsXXX3+tBDIh5P2IQE6ZMgW+vr7qZqNRo0b650WAf/vtNyxYsADnz5/Hhx9+iD59+mD37t3q+adPn6JZs2aoWrWqOpe8Xm5cunfvHuc6v/76q7ohOHz4sLrBkev5+PgY/bciJM2QUoaEENPj7e2tsbW11WTLli3ONnXqVP0x4eHhmipVqmi6d++uKV++vGbgwIFJnjMwMFBKj2rOnj2r2jdu3FDtn3/+WX/M8uXL1b4dO3bo902bNk1TpkyZOH1zc3PThISE6PfNnz9f4+LioomOjlbtxo0ba0aOHKke37p1S72XO3fuxOlP8+bNNWPHjlWPK1WqpJk8eXKyxmbt2rWa7Nmza4KCgl55LiwsTOPs7Kw5cOBAnP0DBgzQ9OrVSz3+4osvNK1atYrzvL+/v3rfvr6++v43aNAgzjE1a9bUjBkzJll9JCQ94JwwIWlI06ZNMX/+/Dj7xDWqQ9zRf/75JypXroxixYph5syZcY4VK1MsWLHkxNWqs4D9/PxQsWJF/XHyeh06K7pSpUpx9j148CDOucXF6+zsrG/XrVsXz58/h7+/v+qLIWLZRkdHo3Tp0nH2i+UrbnJBLNshQ4bgn3/+QYsWLdClS5c4/TKkZcuW6holSpRQ1rRsYuFLf8Rqf/HihTrGEHGVi+UrnD59Gv/991+Clra463X9jH/9ggULvjIOhJgTijAhaYi4QkuVKpXkMQcOHFD/P378WG3yGh0yxypitWjRIri7uysRFvGNP3drb2+vfyxzxAnti+/CNgYRZ1tbWxw/flz9b4hOCN977z20bt0amzdvVkIsLuXvv/8ew4cPf+V8rq6uynUurnA5Vm40ZL5W5rHlWoKcR+afEwpqk2NkbMTlHR8R2oTGxRTjQIipoQgTYkbEapP5ThHZlStXwtvbG//++6+a+3306JGaL5XnJOhK2Ldvn8muLdZkaGioCnwSDh06pAS1SJEirxwrFqhYwmJF6vqSEPJaCfCSbezYsarvCYmwIHPXYjHLJnPaEny2c+dOZQGL2Iq137hx4wRfW61aNaxduxbFixdX5yEks8JPLyFpiLhrAwIC4n7p7OyQJ08eJWoSbCTWY79+/ZRLVlzIYj1+/PHHKspYXL0LFy5U1p2I0qeffmqyvok1PWDAABXQJVHWIoQSfCU3APER927v3r3Rt29f1T8RZYm2luAucfm2b99eBZlJtLIc++TJE+UuLleuXILX/vvvv3H9+nUVjCXvU6KoxUItU6aMspIlCltuTmSfRIdL4Nr+/ftVFLfcqEgQmAh8r1699NHP4saWoDEJbItvrROSUaEIE5KGSNSuoXtUEKG5dOkSpk6dqpb1iCAJcpwIrghLq1at1JytiIrMtYoLWl73448/qqhqU9C8eXO1REiEUG4W5LpJLeGR9c5ffvklRo0ahTt37qgbiTp16qhlU4LcVIg43r59W4ml3FTEn+PWIVavRGPL9cLCwlQ/JEJbljUJX3zxhVo6JS5tEWs5XqzfcePGqefFNS+iPGbMGDVW0n9x28s1E7qJICSjkkWis8zdCUJI+iLrhGWZz4YNGzj0hJgR3jISQgghZoIiTAghhJgJuqMJIYQQM0FLmBBCCDETFGFCCCHETFCECSGEEDNBEU4hc+fOVdl6pAyblHQ7cuQILBGpiCPpAWVdpqT8i7+kRVa4ScpBWeMqmZck+5Guqo4OScUoiR5k7ais95QEEbrUhDqkKo9kYpLxlKxLUvEmoyNrWKVsoCSXkPKDUhpQMlwZImtgZe2sJN2QbFSST1lXplCHJOGQZBeSN1nOI4k6oqKi4hwj6R1lnaxkkpI0mEuXLkVGRvJlSxIP+ZvLJnmpt27dCmsfl8SYPn26+n5JwhMd1jxGkydPVuNhuJUtW9YyxyZdykRYGCtWrNA4ODhoFi9erDl//ryqfJMzZ07N/fv3NZbGli1bNJ999plm3bp1qkLN+vXr4zw/ffp0TY4cOTQbNmzQnD59WvPmm29qPDw8NKGhofpj2rRpo/Hy8tIcOnRIs3fvXk2pUqX01XCEZ8+eafLnz6/p3bu35ty5c6oKUNasWTU//fSTJiPTunVrzZIlS1SfT506pWnXrp2maNGimufPn+uPGTx4sKZIkSKqotGxY8c0derU0dSrV0//fFRUlKZixYqaFi1aaE6ePKnGO0+ePPrKRML169dVVaGPPvpIc+HCBc3s2bNVRaNt27ZpMiqbNm3SbN68WXP58mVV1WjcuHEae3t7NVbWPC4JceTIEU3x4sU1lStX1letsvYxmjRpkqZChQqae/fu6TepIGaJY0MRTgG1atXSDB06VN+W0m/u7u6qXJwlE1+EY2JiNAUKFNB8++23+n1Pnz7VODo6KiEV5MMtrzt69Kj+mK1bt2qyZMmiL4s3b948Ta5cuVRZPx1Sbs6w9F5m4MGDB+q97t69Wz8WIjyrV6/WH3Px4kV1zMGDB1VbfhxsbGw0AQEBcUoKSpk/3Xh88skn6gfJkB49eqibgMyE/I2l5CLHJZbg4GCNp6enxsfHJ07pSGsfo0mTJqkb94SwtLGhOzoF+Xalkoy4XXVImjxpS8Fza+LGjRsqL7LhWOTIkUO553VjIf+LC7pGjRr6Y+R4GTMpz6c7RlInSlk/HZJPWVy7koM4syD5jQ1LFcrnJDIyMs74iEutaNGiccZH8kXryg/q3ntQUJAqZq87xvAcumMyy+dN0llK+s2QkBDllua4xCIuVXGZxv/7coygprVkGkzKXcp0lriXLXFsKMJGIjVd5UfF8I8rSDt+on5LR/d+kxoL+V/mY+IXMBChMjwmoXMYXiOjI4UGZD6vfv36+jq/0ne5sZCbkKTG53XvPbFj5AdFqiBlVKQGsczXyXybVFVav349ypcvb/XjokNuTKSco8QWxMfaPzu1a9dW87OSe13iC+SGX2JGgoODLW5sWMCBEBNZNOfOnTNpqcHMjhScOHXqlPIQrFmzRlU/2r17t7m7lSHw9/fHyJEj4ePjo4IRSVykGpcOCfATUZYCHatWrdKX3rQUaAkbiVSOkTJp8SPxpF2gQAFYE7r3m9RYyP9Sg9YQiVCUiGnDYxI6h+E1MjJS/k8qIUnpvsKFC+v3S99l+kIKJSQ1Pq9774kdI1HHGfkHSawViTitXr26svakKtSsWbOsflx0LlX5XkhkrniGZJMbFKmSJY/FIrPmz058xOqVEplSrtLSvlcU4RT8sMiPitRRNXRFSlvmu6wJDw8P9UE2HAtx5chcr24s5H/5ssiPjg4p3C5jJne3umNkKZTM8+gQC0EsKak1m1GRWDURYHGzynuS8TBEPif29vZxxkfmuWVuy3B8xG1reKMi711+CMR1qzvG8By6YzLb503+5lJykOOiLSMpf3fxFOg2iZuQuU/dY352YpEljdeuXVNLIS3u85OuYWAWtERJIoCXLl2qon8HDRqkligZRuJZChK9KSH+ssnHZcaMGerxrVu39EuU5L1v3LhRc+bMGU3Hjh0TXKJUtWpVzeHDhzX79u1T0aCGS5Qk2lGWKL3zzjtqCYuMrywdyOhLlIYMGaKWZ+3atSvOUooXL17EWUohy5Z27typllLUrVtXbfGXUrRq1Uotc5LlEXnz5k1wKcXHH3+sokDnzp2b4ZeZfPrppypK/MaNG+pzIW2JiP/nn3+selySwjA62trHaNSoUep7JZ+f/fv3q6VGssRIViBY2thQhFOIrCmTD4GsF5YlS7IG1hL577//lPjG37y9vfXLlCZMmKBEVG5MmjdvrtaFGvLo0SMlui4uLmqJQL9+/ZS4GyJrjBs0aKDOUahQISXuGZ2ExkU2WTusQ25G3n//fbU8R77wnTt3VkJtyM2bNzVt27ZVa6Plh0Z+gCIjI1/5O1SpUkV93kqUKBHnGhmR/v37a4oVK6b6Kz9+8rnQCbA1j4sxImzNY9SjRw9NwYIFVZ/l90DaV69etcixYRUlQgghxExwTpgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCCCHETFCEU4Fk/5Hi0/I/4fjws2M6+N3i+FjLZ4frhFOBpGiU0n2SoF7SoRGODz87poHfLY6PtXx2aAkTQgghZoIiTAghhJgJq6snLGX0Tp48qUqF2dik7h5ECkwLd+7cUS4QwvHhZ8c08LvF8cnMnx2pGCZlEatWrapKUyaF1c0JHz16FLVq1TJ3NwghhFg4R44cQc2aNZM8xuosYbGAdYMjtSkJIYQQU3Lv3j1l7On0JimsToR1LmgR4MKFC5u7O4QQQiyU5Ex5MjCLEEIIMRNmFeE9e/agQ4cOcHd3R5YsWbBhw4bXvmbXrl2oVq0aHB0dUapUKSxdujRd+koIIYRYlAiHhITAy8sLc+fOTdbxN27cQPv27dG0aVOcOnUKH3zwAd577z1s3749zftKCCGEmBqzzgm3bdtWbcllwYIF8PDwwPfff6/a5cqVw759+zBz5ky0bt3apH2Ljo5GZGSkSc9JSEbAwcEh1cvzCCGmIVMFZh08eBAtWrSIs0/EVyxiUyErtgICAvD06VOTnZOQjIQIsNzMihiTjElYZDSO3XyCyOgYc3fF6sjr6oiKhXKk2/UylQiLOMYP+Za2LMgODQ1F1qxZX3mNJPE2TOStW8id1DVEgPPlywdnZ2c1V02IpSBJBO7evauWUBQtWpSf7wzIzkv3MWnTefg/DjV3V6ySNyoXxJy3q6Xb9TKVCKeEadOm4fPPP0+2C1onwLlz507zvhFiDvLmzauEWLLH2dvb84+QQbj95AU+/+sCfC7cV+08Lg5wz/mqYUHSlqJuzkhPMpUIFyhQQKUCM0TaUikjIStYGDt2LD766CN9W1KZlS9fPsFjdXPAYgETYqno3NBy00kRNj/hUdH4ee8NzN55BWGRMbCzyYIBDTwworknsjlmqp9okgIy1V+4bt262LJlS5x9Pj4+an9iyFIm2XQkJ5coXdDEkuHnO+Ow/+pDTNh4DtcDQ1S7tocbvuhUEaXzu5q7a8QaRPj58+e4evVqnCVIsvTIzc1NzVeJFSuW62+//aaeHzx4MObMmYNPPvkE/fv3x86dO7Fq1Sps3rzZjO+CEEKM435QGL74+wL+PnNPtfO4OGJ8+3LoWEWbM4FYD2Zdp3Ds2DFVZUI2QdzG8njixImqLcEjfn5++uMlolMEV6xfWV8sS5V+/vlnky9PIlqKFy+OH374IdnDIYlU5AeEkeWEJExUdAx+3nsdzb/frQTYJgvwbr3i2DGqMTpVLUQBtkLMagk3adJELQlKjISyYclrpBQhieV1d86TJk3C5MmTU1RxKlu2bMk+vl69eurGKUeO9AvvJySzcPTmY0zYcA6XArQrNKoWzYkvOlZM1+UwJOORqeaEScKI8OlYuXKl8iT4+vrq97m4uOgfy02PBOS8rsalLorW2IAfCZ6zRiIiIrjuliTIw+fhmLblEtaeuK3auZzt8WnbsuhWvQhsxBQmVg3T5lgAIny6TaxQsYx17UuXLsHV1RVbt25F9erVVZCaZBm7du0aOnbsqNZZi0hLzct///03SXe0nFfc/507d1YR5J6enti0aVOi7mjxZOTMmVOlFZXsZnKdNm3axLlpkGUyI0aMUMfJsrAxY8bA29sbnTp1SvT9Pnr0CL169UKhQoVUPypVqoTly5e/sh72m2++UfnF5T1LjMHUqVP1z9++fVudQ+IPxNqvUaMGDh8+rJ579913X7m+JIQRL4wOeTxs2DC1P0+ePPopkRkzZqj+yDmLFCmC999/X8U+GLJ//371eul7rly51GufPHmiYh9kDAzXtQvSl3feeSfR8SAZk+gYDX4/dAvNvtulF+BetYpg56gm6FGzKAWYKCjCr0EsxxcRUWbZknLVG8unn36K6dOn4+LFi6hcubIShnbt2mHHjh3KvS/iKMU0DOfgE0LWXHfv3h1nzpxRr+/duzceP36c6PEvXrzAd999h99//10V7JDzjx49Wv/8119/jT///BNLlixR4iTR668r5BEWFqZuKCQ+4Ny5cxg0aJASKakRrUOC+uT9TpgwARcuXMCyZcv0iV7kvTdu3FgF/clNxOnTp1Wwnwi3Mfz666/K+pV+S0pVXTaqH3/8EefPn1fPS/CgnFuHBB42b95cLZOTDHByQyTjLt6Jbt26qf8Nb2wePHig3qcEIpLMw2n/p+g8b79yPweFRaGCe3ase78epr1VGbmyMVMZiYXu6NcQGhmN8hPNUyDiwpTWcHYwzZ9oypQpaNmypb4tFqAEt+n44osvsH79eiUAYuElhliJYkEKX331lRIcET8R8cTWXotAlSxZUrXl3NIXHbNnz1aCKda1INHv8ZehxUcsYEMhHz58uLK2JVJeCmlLVrRZs2apc4lVLcj1GzRooB6LIAcGBqo5bxkHQSxmYxFPgFjbhhimUBVPwpdffqmi+ufNm6f2yfFidevaQoUKFfSP3377bXVDIoIs/PHHH8qKN7TCScbl6YsIfLvdF8uO+EHuoV2d7DC6VRn0qVMMtnQ9kwSgCFsJ8sNviFiDEqwlVpa4h8UtLKk/X2cJixWtQ1yukihFrLXEEJerToCFggUL6o9/9uyZSrYiwqnD1tZWWblJWaViLcoNgIiuWLMyHysuXF2SFbH2pS0WZ0KINSpR+DoBTinSz/iIS1+ytMk0gFj1Mq5iuYtHQPon19YJbEIMHDhQTQ3I+5KbDXHpy40Pl61kbGJiNFhz4jamb72ExyERat9bVQthbLtyKhcxIYlBEX4NWe1tlUVqrmubivhRzmJJylIvcRWLFSgZx7p27aoELSniZ1gScUhKMBM6PrVu9m+//VZZujJfrZt/FQtU1/fEsqfpeN3z4lKO38eEKmrFH9ObN2/ijTfewJAhQ9T8s4i8uJsHDBig+iYi/Lpry82BeChkfrhVq1bKrc118BmbC3eDVMKN47eeqHbp/C4q6rl2Caa+Ja+HIvwaRDRM5RLOSMg8plhYOjewWMYiIumJBJHJPK24hRs1aqS3ck+cOIEqVaok2XcJKuvTp49qy03A5cuX9elIxU0sYifz3VJvOiFrXgLMZC47IWtYosJlrtkQsWBfl+Lx+PHjqi+yfl1XKlCs9fjXln4llc9c+iw3GGINS9UwCfAiGY/gsEjM9LmCXw/eVEFYzg62+KCFJ/rV94C9bSrDbeTG9skNIDqBcqo5CgGOLzNqhT4FggMAB2cgZ9HYYwIvAxojKzC55gey5tI+Dn8OPLsN2DkCbh6xxzy6lnCfkiJbXiDbyxuSyFDgyS3Axg7IYzAF9OQmEBlm3Hmlr9JnQfokfZPlmnnLxB7z1B+I0GYjSxZOOYDsBZGeWJ66kGQhQrVu3ToVFCQ3GhLAZGxgkimQ+Vxx34o1XrZsWTVHLJHCSblfpe9r1qzBgQMHVHSxRCSLW1snwk5OTirKWgKiJHCqfv36ag5YrEqxSmVOW9zZEnUs1xYXuQSnubu7qxSozZo1U9a2WKPSlnlZEWVdUpnEkPcgFrO8BxlXw4AtHTL/Lda7RE3LXLH077///lMuaomy1s0Li6di0aJF+mxxJOMgXpJNp+9i6uaLeBCsjWRvX6kgxr9RDgVzpLLgggjR2VXAgTnAw9hlhnHotQIo87IO++VtwPr/ASWbA++siz1mUVMgIm5U/mt5czZQra/2sd8h4M8uQEEv4H97Yo/54y2tYBpD80lAw5f5+wMvAQubANkLAR9diD1mzQDgzjHjzlt3GND65YqH5/eBebUBW0dggsH02JbR2jFKLlX7AB3nIj2hCFspIlwScSsJNuTHX0QrOXm1TY1cV8pH9u3bV80HS6SzLNmRx4kxfvx4XL9+XR0nLl55jQiqzDHrkJsKWQsta6alYpAIrYieIML3zz//YNSoUSrCW+ZtRcDnztV++eS88noRcZnPlXGS/p09ezbJ9yJuZBlXifgWsRXrXkReXqujdOnS6trjxo1Tc+FisdeuXVsf7KbzEHTp0kW5oZNaqkXSn6sPgjFx43kcuPZItT3yZMPnb1ZAo9LGral/hRePgWO/AIcXAiEvRUQExTF2jb8eWwOPjK0D4JwbcMoe95isblor1hjsnAzOa/fyvDletT7Dky4H+wr2BjcmNi/Pq7O4dch1ZL8x2BsU2slio329jJkh4jEw5rwOCYx3GpNFY8p1MJkAWR8q7j1/f38ULlw4znPygyv5qyU9plhTJP0Ra1zWFMsyKInYtlYkqEyipiX63NTwc248smRw9s6rKuVkZLQGjnY2GNa0FAY1LgFHu1TEbohVeXAecPJ3IPKFdl/2wkCdIVqrNL64kkyvM/GhJUzMyq1bt5RlKOt2JaJZlhXJjZC4ZK0RccVL0hPZDJcxEfMgNsr28/dVsYU7T0PVvhbl8mFShwooYoq6s+J2PrpI+zh/JaD+CKBC57jWLrFoKMLErEgAkyzDkTlQ+cGrWLGiWuYj1rA1IvPOIsTi0i5TxiDAhKQ7tx6FYNKm89jlG6jahXJmxeQ3K6Bl+ZfBQMYiMRdXfbTzoQUqavfVfV8bgCXzmyWaaAOLiFVBESZmRVw2EsBEtKR3hDp5lbDIaCzYfQ3zdl1DRFQM7G2z4H+NSmJo01LI6pAK1/POL4B9M4BybwI9ftfucysB9FnLP4MVQxEmhJCX/Of7AJM3ncetR9r52Qal8uDzjhVQMq9LyoKtosJjl7xU7g4c/UUrvBKKQ6uXUIQJIQS4+zQUU/66gG3nA9Rw5M/uiAlvlFdLj4zOVibBVofmAyd+B8p1AN76Sbs/XzlgtG/caGFi9dASJoRYLeJu/mXfDfy444rKEy/5nfvXL46RLUrDxdHIn8c7J4ADs4ELG2ITZcha3+go7ZIfgQJM4kERJoRYJQeuPVRrfq8+0Ca1qFXcDVM6VUDZAtmND7YS8b25N3Z/yWZAvREMtiKvhSJMCLEqHgSFYeqWi9h46q5q53FxwNi25fBWtULJdz3LXO+ZVcDBOdosULpEFBW7AvWGx0Y/E/IaKMKEEKsgKjoGvx28hZk+lxEcHqXiot6pUwyjWpVBjqzJXJcb+gQ4thg4/JM2VaLgmB2o/i5Qe7A2rzMhRpDKLOPEkpCatfHr4UohgaQQy2HDhg2pvrapzkNIQkiFow5z9mPK3xeUAHsVyYlNQxtgSseKyRdgYd0gYMcUrQDLet9WXwIfngNafUEBJimClrAFIMUCpHDAtm2vJirfu3evymF8+vTpOLWAk4NUN4pfri+1SA1jEVupSmSI1DSWYgyEmJJHz8Px9bZLWHXstmqL4I5pUxY9axaBjU0yXM93TwI5igDZtMU1UHMgEHRP63Ku+BYzW5FUQxG2AKQykCT8l3yl8fOULlmyBDVq1DBagHUl/dKLAgUKwBqROsNSUIKYlpgYDZYf9cM323zxLFRbeq9HjSIY07Ys3LIlc7y3fAIc+QloPAZoOk67z7OlduMaX2Ii6I62AKSQvAimpH80RGoEr169Won0o0ePVKWeQoUKqcpDUk5v+fLlSZ43vjv6ypUryqqW4hZSdcjHxyfBqkhSKUiuUaJECVWNSKx0QfondXTFKhf3s2y6Psd3R0vFIikpKFWGcufOrSolyfvRIbWQpcLQd999pyokyTFDhw7VXyshrl27puoQSw1jFxcX1KxZU6XINETyV8t7kExejo6OqjzhL7/8on9eyiHKeGfPnh2urq5o2LChOm9C7nxB+ih9NRxTKUwhlZXkHPK+XjduOv766y/VZxl/qXylqwU9ZcoUle4zPlKTWc5jbZy9/Qyd5x/AZ+vPKQEuVzA71g6pi6+7Vk5agCXYKuJlEQWhWF1tsFVYbHUuJb4UYGJCaAknF2MKQ+uQslq69YGyVjA6XFtyy3CtYGLndUi+G1hK9smPugjaZ599po/wFAGOjo5W4isCVr16dfVjLz/+UibvnXfeQcmSJVVJveRUN3rrrbeUgB0+fFiVDYwvOIIIk/RDavOKkA4cOFDtk7KAPXr0UHV5xW2uEz8p2xefkJAQVU5QavmKS/zBgweq0P2wYcPi3GhIHV4RYPn/6tWr6vwiPHLNhJAxkNKFU6dOVQIrtXrFle/r64uiRbUF0WUcDx48qKoXSWlCKSbx8OFD9dydO3fUTYiI7c6dO9U4SspNKYVoDHLjICUWJ02alKxxE+TvJaIrf1/pt1jQW7ZsUc9JqUW5uZGxEpEWpD7ymTNnVM1oa+HZi0h8948v/jh8SyWkknW+o1qVVsFXdrY2yQu2kupFDT7U7pf0kiPPcK6XpC0aK8Pf319KN6r/4xMaGqq5cOGC+v8VJmU3fju3Lvb18lj2LW4X97xfeyT8WiO5ePGiel///feffl/Dhg01ffr0SfQ17du314waNUrfbty4sWbkyJH6drFixTQzZ85Uj7dv366xs7PT3LlzR//81q1b1TXXr1+f6DW+/fZbTfXq1fXtSZMmaby8vF45zvA8Cxcu1OTKlUvz/Plz/fObN2/W2NjYaAICAlTb29tb9S8qKkp/TLdu3TQ9evTQGEOFChU0s2fPVo99fX1VP3x8fBI8duzYsRoPDw9NREREgs/HHz+hY8eOqq86pM+dOnV6bb/ij1vdunU1vXv3TvT4tm3baoYMGaJvDx8+XNOkSZMEj03yc54JiYmJ0aw55q+pNuUfTbExf6ttxPITmvvPXvP+Ht/UaLaM0Wi+LBj7vfupiZwwvbpOrFBn4kNL2EIoW7Ys6tWrh8WLFytLTSxDCcoSV6UgFvFXX32FVatWKYtOLClxvYr7MzlcvHhRuWjFUtMhlmp8Vq5cqaxIcdGK5SlWoliMxiDXEivUMCisfv36yhoXq1WscUHq7draxibUF6tYrMjEkP5IYJhYlRIIJn0LDQ2Fn5+fel6CxeR8UlYxIeR5cT/b26euzJzM0Rs7bnLtxCx8QZ4Ti3jGjBmqMtWyZcswc+ZMWDqXAoIwccN5HLn5WLVL5XPBlI4VUK/ky0CqxIKtJLnGeclsFa3dl6/CyzKCb9HdTNIVinByGadd2G+0O1pH2Q7ac4g72pAPEhcNY5G53+HDh2Pu3LkqIEtczTpB+fbbbzFr1iw1xyvzwSJw4k4WMTYV4sbt3bu3co2KO1lczStWrMD333+PtCC+GIobXoQ6MaRcosxjiztY5nplvrlr1676MZB2UrzueRE/rVEfS0Jz1PEjzpMzbq+7trjVxcW+fv16Fegl15X3Zqk8D4/CDz6XseTATUTHaJDV3hYjW3iif30PONjZJJLZ6l/gwI9xM1tJ+UDJbCUZrjjXS8wARTi5GDFHmyAyN6ybHzbleQ3o3r07Ro4cqawgmTccMmSIfn5Y5i4lKKlPnz6qLWJ1+fJlFWCVHKS+r7+/v7IgxeIUDh06FOeYAwcOoFixYmreUsetW7fiHCMCIVb5664l86MyN6wTLOm/iFxqauzKOSRIShfQJBanYelAuTmRcdm9ezdatGjxyuslwvzXX39VApeQNSzBcTI+OuR9yhx406ZNk+xXcsZNrr1jxw7069cv0bgAb29vdfMlY9yzZ8/XCndmRG5yNp+9hy/+voD7QeFqX5sKBTChQ3lV7zfBYKuzq7WWry6zVRZboGIX7TKjgsavGiDElDA62oKQiF8JTho7dqwSA8OoXE9PT2UFyg++uHv/97//4f79lxl/koGIkkTvyg+9RDeLq9tQNHTXENeuWHHiVhX3qlhmhkh0sAQ7iXtVAp7EJR4fsQolAliuJSImgVdi4Usgmc4VnRKkfxKoJNeW9/D222/HsZylb3JNcetKpLb0c9euXcqFL0hgWFBQkBK4Y8eOqWjx33//XbnIBYnmFle3bJcuXVI3QU+fPk1Wv143bhLEJdHs8r/8/cTt/vXXX8c5RoLXJGBMAt/kPVga1wKf451fjmDYspNKgIvldsbSfjWx4J3qCQuwsLgNsHGoVoAdXIC6w4CRp4EuiyjAJENAEbYwxCX95MkT5dY0nL8dP348qlWrpvbLnLGsy5XlM8lFrFARBplDlWhq+cGXKGND3nzzTXz44YdKrCRKWQQ//hIZWc/cpk0bZR2K5ZjQMimZp96+fTseP36son3Frdq8eXPMmTMHqUHmSyUhiMydi/tWxkLGxJD58+er673//vtqnl3mWsUiF2QZlIicWNDi5pdo80WLFumtYhE+EXGJsJbnZanR66zg5I6b/M0k2n3Tpk3qGBH8I0eOvCLm8t6k37Vr14alEBoRje+2+6LND3uw7+pD5W7+oIUntn/QCE3K5It78FM/7UoEHRU6Aa4FgZZTgA/PA62nAjmLpPt7ICQxskh0FqwISWghAUbiWo2f2CIsLExZPx4eHsoSIyQzIV9lEWK5gfjoo48SPS4zfc59LtzH5E3ncedpqGo3LZMXk9+sgGK5E5jG2fIxcPQXrZUr7mYhMlTrfrZjQhSSMXQmPpwTJsQCCAwMVO7sgICAROeNMxP+j18o8d1x6YFqi7t5YofyaFU+f2ylI539oGs759ZGO/sfiRVh1u8lGRyKMCEWQL58+VQWrYULF2bqHNzhUdH4afd1zP3vKsKjYmBvmwXvNSyB4c1Kwdnh5c9VVIQ22ErKCDafBJRpo91faxBQpi1Q0Mus74EQY6AIE2IBWMKs0p7LgZi06TxuPNTOwdcrmVtVOZK1v4rQp8DxJdrMVsEvo9CPLooVYWc37UZIJoIiTAgxK/eehaolR1vOBqh2PldHjH+jPDpULqh1PUuw1aEFwIlfgYiX+cMl2Erq90odX0IyMRRhQohZiIyOwZL9N/DDv1fwIiIatjZZ4F23OD5s6QlXJ3vg3mnt+t5z6+JmtlJlBLsw2IpYBBThBEgq6xIhmZ2M4Lo+dP0RJm48h8v3tZZtjWK5lOu5fEFX4OoObWarG7vjZbYaDpRszsxWxKKgCBsgmYZkPezdu3fVGlZp6yMxCbEQAZZIavlcpzYHdkp4EByGaVsuYf3JO6otpQXHti2LLtUKw0as3YVNgHun4mW2GsZgK2KxUIQNEAGWtZOSbUqEmBBLRARY1i4aFr9IayS/8x+HbqmkG8HhUWpV0du1iuLjpoWRM6cumtsOyFceeHRVO9crc75MrEEsHIpwPMT6ldqyUsXmdTmOCcmMiAWcngJ8wu8JJmw4h/N3g1S7UqEc+LJjBXhd+h6YtxTovw0oUFF7cItJQJtpQNac6dY/QswJRTgBdK46c7jrCLEUnoRE4Jvtl7D8iL9qZ3eyw8dtyioLWIKwcMgPiAgGzq2JFWHXAubtNCHpDEWYEGJSYmI0WHXMH19vu4QnL6SUowbjytzDu/gLDp4zARFgofGnQNW+QKnm/AsQq4UiTAgxGefuPMOEjedw0u8p7BGFYW4n8L7DVjjf0laawsG5wBsztI/zl9duhFgxFGFCSKoJCovEjH8u47eDN+GiCcFwh/8wOKsPsr0IBF5IsIULUM0bqDOEo02IARRhQkiqljxtOHUHUzdfgsPzOxhrtw19HHYha8wLIDxeZisGWxHyChRhQkiKuHw/WEU9P795Ap/ZbcabTgdhixjIP7XUSGW26srMVoQkgQ3MzNy5c1G8eHFV11QKkccvVG5IZGQkpkyZgpIlS6rjvby8sG3btnTtLyHWTkh4FKZtuYjus7Zj+O1R2Ow4Dp1t92sF2KMx0HstMOQAUOVtCjAhGdkSXrlypSo+vmDBAiXAP/zwA1q3bg1fX19Vmi0+48ePxx9//IFFixahbNmy2L59Ozp37owDBw6gatWqZnkPhFiT63nr2Xv4YvNF3HsWBsAJhV2joImwRZaKbwF1hwHuVczdTUIyFVk0ZkwkK8Jbs2ZNzJkzR5+zuUiRIhg+fDg+/fTTV453d3fHZ599hqFDh+r3denSBVmzZlXinBxu376truHv76+yBhFCXs+N+09waNmXqPFkK7pETEYOtzz4/M0KaJb9nrZ8YM6iHEZCUqAzRlvC4jru378/3n33XZVZKqVERETg+PHjGDt2bJy0kS1atMDBgwcTfE14eLhyQxsiArxv375EryOvkU1HcHBwivtMiFURFYG7z6OxeN8NFfX8l+02eNrcwaxyF1D37QlwspesW/nN3UtCrGtO+IMPPsC6detQokQJtGzZEitWrIgjcsnl4cOHKi1k/vxxv8TSDgjQ1hWNj7iqZ8yYgStXriir2cfHR/VFcj0nxrRp05AjRw79Vr481yUSkihB94BjSxC8+C2EfVUMb3zzF37edwMR0RpszjcIgc1nomnvsS8FmBBiFhE+deqUCqAqV66cch0XLFgQw4YNw4kTJ5CWzJo1C56enmo+WHI8yzX79eunLOjEEEv72bNn+u3ChQtp2kdCMhUyGxVwFtj9DTQLmwIzygJ/fwBXvx1winmBWjiPuiVyY0m/mvhw6AjkbdgfsHM0d68JsRhSHJhVrVo1tX3//feYN28exowZg/nz56NSpUoYMWKEEsekygDmyZNHJZG/f/9+nP3SLlAg4fyxUl5ww4YNCAsLw6NHj9Qcscwdi1WeGI6OjmrTERSkTSJPiNUSFQ7c3Af4btVuQbfVbt239VRMSeyIqY7wUm0wtHlzVCrCYgqEZDgRluVC69evx5IlS5RbuE6dOhgwYICakB43bhz+/fdfLFu2LNHXiyVbvXp17NixA506dVL7xMUsbbFwk0LmhQsVKqT6sHbtWnTv3j2lb4MQ6+H8eu12dQcQ8Vy/OwwO2BtdCf/GVMNB2+poUdML/eoXRxE3Z7N2lxBrwGgRFpezCO/y5cuVG7hv376YOXOmchHrkGVDEvX8OmR5kre3N2rUqIFatWqpJUohISHKihbk3CK2Mq8rHD58GHfu3EGVKlXU/5MnT1bC/cknnxj7NgixfB5fB9wMvERn1wCX/lYPg+3zYFuEF7ZGVsWBmApwdc2uhHdcrWLI4czqYYRkWBEWcZWALHE9iwWbULk/Dw8P9OzZ87Xn6tGjBwIDAzFx4kQVjCXiKsk3dMFafn5+ceZ7xQ0ta4WvX78OFxcXtGvXDr///jty5qS7jBA90VHAggZA4EVg2HEgTynt96lYF1wKdMP8gNI4FVYcGtjAM58LpjQqgY5V3OFox2ArQjL8OuFbt26hWLFiyKxwnTCxKMKCgKv/Ag8uAs0+i93/65vArQPQvLUIex0aYNHe69h75aH+aQm2GtSoBBqXzgsbXWlBQkjGXyf84MEDZbVKog1DxFUsgVbiWiaEpCFP/QDfbYDvFm2AVUzkSzfVAMBVG9QY0XYmtt2IxLx/H+BSgDYVrK1NFrSrVBADG3qgcmF6jwjJCBgtwpKtSuZg44uwzNF+/fXXSowJISYkJga4exK4/DKa+f65uM/nLgWUaauWG0lJwRVH/LB4300EBElqScDZwRY9ahZB//oeDLYiJLOLsKyzlaVJ8ZHczVyDS4iJiHgB3NitFd3L24DnBkv5stgAResCpdtoxTePJ+4+DcXSfTex7PAZPA+PUofldXXEu/WKo09tBlsRYjEiLGtuZS1v/LW5krXKzo6VEQlJNRKmMaemfv2uwsEVKNUcKNMO8GypzdcsN8V3g7Bo5Sn8dfouomK04R0SbDWQwVaEZAqMVs1WrVqpLFQbN25UaSCFp0+fqrXBEjVNCDEysOrIT8Dt40Cv5YAkuJGteAPg1n6tpStbsQb6soASS7nvSiAW7okbbFWnhBv+16gkg60IsWQR/u6779CoUSMVIa0rHyhpLGVZkSwXIoQkQVSE1sLVrd+1dQD2zgAiXwABZ4CCXtr97b8HHLJpBfklkdExyuIV8b0UoC1EIoHN7Su7M9iKEGsRYUmecebMGfz55584ffq0qmIkyTV69eqV4JphQqyeF4+BKz7awCrJVpXdHRj6MoDR3gloNBpwzg3kKBI7VI4u+ofBYZFYfsQPS/bffFnHl8FWhFgKKZrEzZYtGwYNGmT63hBiKTy8GhvN7HcI0ETHPvfCCUqYX87rouGoBE9x71moEt7lh/0QHC/YqnftosjprHVPE0IyLymOpJJIaMloJXWBDXnzzTdN0S9CMl+WqttHYosiPLoS9/l8FV7O77YD3KtK8exETyXBVj/vvY5NBsFWpfK5YFDDEuhYlZmtCLFqEZaUkZIb+uzZs6pKki7hlq5iktQIJsRqCA8GNo8GrvwDhD6O3W9jrw2uEuGVpUS5ks4yp4Ktrj58Jdiqtocb/te4BJqUzsfMVoRYIEaL8MiRI1VuaKl2JP9LXWEpKzhq1CgVtEWIRfPUH3h0FSjZVNt2cAFu7tUKsFNOoHRrrfCWbA44ZX/t6STY6u8zEmx1AxfvBemDrbSZrUrAi2UECbFojBbhgwcPYufOnaoesBRXkK1Bgwaq0pHUET558mTa9JQQcyPLiH5uBmR1Az6+CtjYaqOX20zXBlYVqQ3YJu8rJcFWK474Y/H+G/pgq6z22sxWAxowsxUh1oLRIizuZldXV/VYhPju3bsoU6aMWrLk6+ubFn0kJH2JDAWu79YGVrkWBJp8qt0vy4ec86gMVQgJ1OdpRvnkx0FIsNXS/ZLZKjbYKo+LoyojyGArQqwPo0W4YsWKammSuKIlf/Q333wDBwcHLFy48JUsWoRkGp4/0KaHlKCqa/8BUaHa/bJsqPEYrcUrVu4HZwEH44vdi6tZKhltOhUbbFUybzZVyahjlUJwsmcZQUKsEaNFWOr5hoSEqMdTpkzBG2+8gYYNGyJ37txYuXJlWvSRENMjAYUPLsRGM985Ljtjn89eODZblRyrS5phhABLsNX+q4+wcO917LkcGCfYSsS3aRkGWxFi7Rgtwq1bt9Y/LlWqFC5duoTHjx8jV65c+ghpQjJstipJBaks3i3akoCGyNIhWUIkwpu/YpxsVcYgwVabz9xTkc4XDIKt2lYqqJYZMdiKEJIiEY6MjFQZsiRNpbildbi5vUw6QEhGLAOoW5P7zB/4vVPsc3ZOgEfj2GVE2Qum6lISbLXyqD8W77uBuwy2IoSYWoQlLWXRokW5FphkfPyPADumANnyAN2WavflLqkthOBWXGvxlmiizc+cSgKehWHJ/hsMtiKEpL07+rPPPlMVk6RYAy1gkiGIiQZuH9Wu2S3w0kNja69dv2ufTeuGflmBCP02m+yylwKClMuZwVaEkHQT4Tlz5uDq1atwd3dXy5Ikj7QhJ06cSHFnCDEqU9W1nYDvNuDKduDFI8DrbaDzfO3zBasA7Wdoa/DqBNgEMNiKEGJWEe7UyWBOjZD05sJG4MRvwI09QLRB3nKnHICjdv26QoKqag4w2WWTCraSzFZVmNmKEJIeIjxp0qSUXIeQ1CfQ2PIxcNKgZnUuj9ho5qJ1tC5oE5NUsFX/+h4omtv4NcOEEJLqKkqEpGtZwNXewP1zYuIC9YYDVfsAeUqneBlRsoKtDrwMtgqLzWz1br1i6F27GHJlYxlBQogZRFhyRSe1HphVlIhJObcO2DQCiAgGsuUFuvysjWpOIyTYatGeG9h0+g4io2MzW4nLuVNVZrYihJhZhNevX//K2mEp2vDrr7/i888/N2XfiLVzaAGwbYz2cbH6QJdfUr2WN7FgqwPXHqn53t0Gma1qSWarhiXQrCwzWxFCMogId+zY8ZV9Xbt2RYUKFVTaygEDTBcMQ6yccm8Ae74BqvUFmo5PdoUiY4KttpzVBludv2sQbFWxIN5r6IGqRXOZ9HqEEBIfk/2q1alTB4MGDTLV6Yi1EugL5C2jfZyjMDDsGOBs2oxsz8OjsOKIH5bsv4k7T0P1wVbdaxRG/wYeKJY79Qk8CCEk3UQ4NDQUP/74IwoVKmSK0xFrRIok+EwEDswGei4DyrbT7jehAN8PClP1e+MGWznAu25x9KnDYCtCSCYQ4fiFGmQ+LTg4GM7Ozvjjjz9M3T9iLchnKkaEUQPcPRkrwibANyBYlRHceCo22KrEy2Crzgy2IoRkJhGeOXNmHBGWaOm8efOq2sIi0IQYRXRU7Fxvi88Bz5ZAyWapHkS5OTx47RF+ih9sVVxbRpDBVoSQTCnC7777btr0hFhfvudd04BbB4G+G7VCLOklUynAumArsXzP3YkNtmpTsYCyfBlsRQjJ1CK8ZMkSuLi4oFu3bnH2r169Gi9evIC3t7cp+0cskeD7wNoB2gILwuWtQLkOJg+2crK3QY8aRRhsRQixHBGeNm0afvrpp1f258uXT0VHU4RJkkjO5zUDgJAH2gpHHWalSoAl2EqE98/DtxhsRQixfBH28/ODh4fHK/ulopI8R0iCxMQA+74H/vsK0MQAecsB3X8D8pZO0YAx2IoQYpUiLBbvmTNnULx48Tj7T58+jdy5c5uyb8RSCHkErBsIXNuhbVfpDbT7DnAwvvjB8VtPMHvnFezyjRtsNbBRCTRnZitCiKWLcK9evTBixAi4urqiUaNGat/u3bsxcuRI9OzZMy36SDIzfoeBNf2AoDuAXVag/Xfa4gtGEhOjwbxdVzHD5zJiNAy2IoRYqQh/8cUXuHnzJpo3bw47O+3LY2Ji0LdvX3z11Vdp0UeSWZNvHJwD/DtZu/43tyfQ/VcgfwWjT/U4JAIfrDyFPS+XGnWq4o4PW5ZmZitCiPWJsIODg8oR/eWXX+LUqVPImjUrKlWqpOaECVGEPgU2vA/4bta2K3bRBmA5uho9QMduPsawZScREBSmop2ndKyI7jWKcKAJIdadttLT01NthLyCjS3w0BewdQDaTAdq9De67q8k2/h57w1M33YJ0TEaleFqXu9qKFsgOwecEGK9ItylSxfUqlULY8a8LDH3km+++QZHjx5V64WJlbqfBRFbsXi7/w5ERwDuVYw+1bMXkRi95jR8LtxX7Te93PHVW5Xg4mjaKkqEEGJubIx9wZ49e9Cu3at5fdu2baueI1ZIWJA2+OrQ/Nh9+cunSIDP3H6K9rP3KgF2sLXBl50qYlbPKhRgQohFYrRp8fz5czUvHB97e3sEBWnTBBIr4+JfwPn1gO82oHJ3IFseo08h7uffD93Cl39fRER0DIq6OSv3c8VCOdKky4QQkiktYQnCksCs+KxYsQLly5c3Vb9IZqLK20DtIYD3phQJcHBYJIYtP4mJG88rAW5dIT/+Gt6AAkwIsXiMtoQnTJiAt956C9euXUOzZtpk+zt27MCyZcuwZs2atOgjyWhEhAC7pgONRgNOObTzwG2np+hUF+4GYeiyE7jxMAR2Nlkwtl059K9fPE6lLkIIsVSMFuEOHTpgw4YNak2wiK4sUfLy8sLOnTvh5ma6AuwkgxLoC6zyBgIvAk/9tGt/U4C4n1cd81fWb3hUDNxzOGFO72qoVpTlMAkh1oPR7mihffv22L9/P0JCQnD9+nV0794do0ePVmJsLHPnzlUpMJ2cnFRN4iNHjiR5/A8//IAyZcoo8S9SpAg+/PBDhIWFpeRtEGM5swpY2FQrwC75gVoDUzSGLyKiMGr1aYxZe1YJcNMyebF5REMKMCHE6kjxmg+JhP7ll1+wdu1auLu7Kxe1CKoxyNzyRx99hAULFigBFoFt3bo1fH19VY7q+IjL+9NPP8XixYtRr149XL58WdU3FtfljBkzUvpWyOuIDAO2jQGOL9W2PRoDXX4GXF79G72Oqw+CMeSPE7jy4Lmq8zu6dRkMblQSNtIghBArwygRDggIwNKlS5X4SiS0WMDh4eHKPZ2SoCwRzoEDB6Jfv36qLWK8efNmJbIitvE5cOAA6tevj7ffflu1xYKWXNaHDx82+tokmTy6Bqz2BgLOyiJgoPEnQOMx2oQcRrLh5B2MW38WLyKikc/VET/2qoo6JVj0gxBivdgYMxcsbmCpoCQW6927dzF79uwUXzgiIgLHjx9HixYtYjtjY6PaBw8eTPA1Yv3Ka3Qua3GFb9myJcF1y8QEXNgILGyiFWDn3ECftUDTcUYLcFhkNMauO6vyP4sA1y+VW7mfKcCEEGsn2Zbw1q1bVfWkIUOGmCRd5cOHDxEdHY38+fPH2S/tS5cuJfgasYDldQ0aNFCBPVFRURg8eDDGjRuX6HXEUpdNR3BwcKr7bvFERQA+E4HDL5NvFK0LdF0MZHc3+lQ3H4bg/T9P4MK9IBVEPaKZJ0Y094Qt3c+EEJJ8S3jfvn1KwKpXr67mb+fMmaMEMT3ZtWuXisqeN28eTpw4gXXr1in3tVR2Soxp06YhR44c+o1rmV+DRDwvaRMrwPVHAt5/pUiAt569hzdm71MCnDubA37rX0tVP6IAE0KIliwaMSmNQCKiJaBK5m3FLSzWrMzt9u/fX9UYNsYd7ezsrJY5derUSb/f29sbT58+xcaNG195TcOGDVGnTh18++23+n1//PEHBg0apDJ5iTv7dZbwnTt3lBD7+/ujcOHCxrx162DlO8DFTYBTTqDzAqBMW6NPEREVg2lbL2LJ/puqXbN4LszuVQ0FcjilQYcJISRjcfv2bbV6Jzk6Y/QSpWzZsinBFcv47NmzGDVqFKZPn66imd98881kn0dSX4pVLYk+dEhdYmnXrVs3wde8ePHiFaG1tdXOTyZ2L+Ho6Ijs2bPrN2NuFKySdt8BZdoB/9uTIgG+/eQFuv10UC/AgxuXxPKBdSjAhBBiqnXCOiRQS6onieovX77c6NfL8qRFixbh119/xcWLF9V8s1jaumjpvn37YuzYsXGCw+bPn69SZN64cQM+Pj4qg5fs14kxMZKge8Dhn2LbrvmBXsuBXMbXh95x8T7a/7gPp/2fIkdWe/ziXQOfti0LO9tUfcwIIcRiMUltOBFAcSkbupWTQ48ePRAYGIiJEyeq5U9VqlTBtm3b9MFafn5+cSzf8ePHqzXB8r+4lfPmzasEeOrUqaZ4G9ZH2DPgp0ZAyANt9HOlrik6TVR0DL79xxc/7b6u2l5FcmLu21VROJeziTtMCCFWPidsTb56q2DHF8Dl7dr0k7lLGv3ygGdhGLH8JI7cfKza/eoXx9i25eBgR+uXEGKd3DZCZ1gl3dp4HghEhQE5i2jbTcZqCzHYZzX6VHuvBOKDFafwKCRC1fv9pmtltKtU0PR9JoQQC4UibE3c3A+s6Q+4FgAG/APYOQK2dtrNCKJjNJi14wpm77wC8aOUL5hd1f4tnidbmnWdEEIsEYqwNRATAxyYpXU9a6K15QdDAoEcxrvjA4PD8cHKk9h/9ZFq96pVFJM6lIeTPQPjCCHEWCjCls6Lx8D6/wFX/tG2K/cE3pgBOBhvtR6+/gjDl5/Eg+BwODvY4qvOldCpaiHT95kQQqwEirAl438UWP0uEHQbsHMC2n4DVOsLlT/SCGJiNFiw5xq+2+6LGA3gmc8F8/tUQ6l8XHNNCCGpgSJsichE7aH5gM8EICYKcCsBdP8NKFDJ6FM9CYnAR6tO4T/fQNV+q2ohfNm5Ipwd+NEhhJDUwl9SSyP0KbBxKHDpb227fCfgzdmAU3ajT3XC7wmG/XkCd5+FwdHOBlM6VkD3GkXUWm1CCCGphyJsSdw9pa39++QmYGMPtP4KqDXQaPezLB1fvP8mpm25iKgYDTzyZMPct6uhvLvxQk4IISRxKMKWwo29wB9dgOhwIEdRoPtSoFB1o0/zLDQSn6w5je3n76t2+0oFMb1LJbg62adBpwkhxLqhCFsKhWsAeTyBHEWATvMAZzejT3HuzjNV+9fv8QvY22bBhDfK4506xeh+JoSQNIIinJl5fB3IWRyQ/NqS8Urq/mbNlSL385+H/TDlrwuIiI5B4VxZlftZckATQghJO5jgN7NyeiUwrx6w9/vYfWL9GinAz8OjMHLFKYzfcE4JcIty+bF5eEMKMCGEpAO0hDMrsvQoKhTwP6zNiBWvznJyuBQQpNzP1wNDYGuTBZ+2KYv3GnrQ/UwIIekERTgzERMN2LxMD1m1t9b1XLp1igR49TF/TNh4DmGRMSiQ3Qlz3q6KGsWNn0cmhBCScuiOziycWwvMqwuEaHM2K8q2ixXlZBIaEY2PV5/Gx2vOKAFuVDovNo9oQAEmhBAzQEs4oxMVDmwfBxz9Wds+NBdoPjFFp7oW+BxD/zyBSwHBsMkCfNSyNN5vUgo20iCEEJLuUIQzMo9vaHM/3zulbTccra3/mwI2nb6LsWvPICQiGnlcHPFjryqoVzKPaftLCCHEKCjCGZWLfwMb3gfCnwFZ3YC3FgKeLY0+TVhkNL7cfAF/HPJT7Tol3PBjr6rI5+qUBp0mhBBiDBThjEZ0JPDvZODgHG27cC2g25IU1f71e/QC7y87jnN3glR7eLNSGNncE3a2DAUghJCMAEU4I/HsNrC6H3D7iLZddxjQYjJga3zKyO3nAzB69WkEh0Uhl7M9ZvaogiZl8pm+z4QQQlIMRTijcMUHWDcICH0MOObQpp4s94bRp4mMjsHXWy/h5303VLt6sVyY3asq3HNmTYNOE0IISQ0U4Yyw9ve/qbGZrwpWAbotBdw8jD7VnaehGLbsBE76PVXtgQ098EmbsrCn+5kQQjIkFGGzkwW4f177sOZ72vKDdo5Gn+U/3wf4cOUpPH0RiexOdviumxdaVShg+u4SQggxGRRhc6HRaPM8S7arTvOBm3uB8h2NPk1UdAxm/nsZc/+7ptqVC+dQxReKuDmnQacJIYSYEopweiN5nsX1/OQm0HGOVoil8EIKBPhBUBiGLz+Jwzceq3bfusXwWftycLQzLosWIYQQ80ARTm/unwV2fQVoYoAqvYDiDVJ0mgNXH2LEipN4+DwC2RxsMb1LZXTwcjd5dwkhhKQdFOH0pqAX0PILbfGFFAhwTIwGc/67qlzQ4tEuW8AV83pXQ4m8LmnSXUIIIWkHRTitEaU8OBfwbAXkLa3dV29Yik716Hk4Plh5CnuvPFTtHjWK4POOFeBkT/czIYRkRijCaUnoE2D9EODyVuDkH8CgXYB9ytJFHr35GMOXnURAUBic7G3wZadK6Frd+CxahBBCMg4U4bTiznFt8YWnfoCtA1B7UIqWHon7edHe6/hmuy+iYzQomTcb5vWujjIFXNOk24QQQtIPinBauJ+PLNKWH4yJBHIVB7r9CrhXMfpUT19EqNST/158oNodq7jjq86VkM2RfzZCCLEE+GtuSsKCgE3DgQsbtO1yHYCOcwGnHEaf6pT/U1X7V7JgOdjZYHKHCuhVqwiyyJImQgghFgFF2FQEnAVWeQOPrwE2dkCrL4Hag7XrgI1Ao9Hg1wM3MXXLRURGa1Ast7NKvlGxkPFCTgghJGNDETaF+/nEb8DWT4CoMCB7YW3u5yI1jT5VUFgkPl17BlvOBqh224oF8HXXysjuZHwVJUIIIRkfinBqiAgB/v4IOLNC25ZlSJ1/0mbAMpLzd58p9/PNRy9gb5sF49qVw7v1itP9TAghFgxFODUcXqAV4Cy2QPMJQL2R2lzQRrqfVxz1x6RN5xERFYNCObNizttVUbVorlR1jRBCSMaHIpwa6g4H7pwA6rwPFK9v9MtDwqMwfsM5rD95R7Wblc2HGd29kNPZIVXdIoQQkjmgCKdq9ByAnn+m6KVX7gdjyJ8ncPXBc9jaZMHHrctgUMMSsLFh9DMhhFgLFGEzsO7EbXy2/hxCI6ORP7sjZveqhloexs8jE0IIydxQhNORsMhofP7XeSw/4q/aDUrlwQ89qyCPi/GZtAghhGR+KMLpxI2HIXj/zxO4eC9ILR0e2dwTw5t5Klc0IYQQ64QinA5sPnMPY9aewfPwKOTO5oBZPauigWee9Lg0IYSQDAxFOA0Jj4rGV5sv4teDt1Rb5n1n96qK/NlTVkmJEEKIZUERTiP8H7/AsGUncPr2M9Ue0qQkRrUsDTtb49YRE0IIsVwowmmAz4X7GLXqFILCopAjqz1m9vBCs7L50+JShBBCMjEUYRMSGR2D77b74qc911W7SpGcKvtV4VzOprwMIYQQCyFD+Ebnzp2L4sWLw8nJCbVr18aRI0cSPbZJkyYqn3L8rX379jAn956FotfCQ3oB7l/fA6v+V5cCTAghJONawitXrsRHH32EBQsWKAH+4Ycf0Lp1a/j6+iJfvnyvHL9u3TpERETo248ePYKXlxe6desGc7HnciA+WHkKj0Mi4Opoh2+7VUabigXN1h9CCCGZA7NbwjNmzMDAgQPRr18/lC9fXomxs7MzFi9enODxbm5uKFCggH7z8fFRx5tDhKNjNJjxjy+8lxxRAlzBPTv+HtGAAkwIISTjW8Ji0R4/fhxjx47V77OxsUGLFi1w8ODBZJ3jl19+Qc+ePZEtW7YEnw8PD1ebjuDgYBP0HHgQHIaRy0/h4PVHqt27dlFMeKM8nOxtTXJ+Qgghlo9ZLeGHDx8iOjoa+fPHjRyWdkCAtrB9Usjc8blz5/Dee+8lesy0adOQI0cO/SbWtinwfxyKozcfw9nBFrN6VsHUzpUowIQQQjKXOzo1iBVcqVIl1KpVK9FjxMp+9uyZfrtw4YJJrl29WC5807UyNg1rgI5VCpnknIQQQqwLs7qj8+TJA1tbW9y/fz/OfmnLfG9ShISEYMWKFZgyZUqSxzk6OqpNR1BQEEzFW9UKm+xchBBCrA+zWsIODg6oXr06duzYod8XExOj2nXr1k3ytatXr1ZzvX369EmHnhJCCCEWuERJlid5e3ujRo0ayq0sS5TEypVoaaFv374oVKiQmtuN74ru1KkTcufObaaeE0IIIZlchHv06IHAwEBMnDhRBWNVqVIF27Zt0wdr+fn5qYhpQ2QN8b59+/DPP/+YqdeEEEJI6smi0Wg0sCJu376NIkWKwN/fH4ULc06XEEKI+XQmU0dHE0IIIZkZs7uj0xsJ/BLu3btn7q4QQgixQHT6otObpLA6EdYth0pqbTEhhBBiCr0pWrRoksdY3ZxwVFQUTp48qQK/4gd8GYukwJQMXJIAxNXV1WR9tDQ4Thwrfq74/bOm36qYmBglwFWrVoWdXdK2rtWJsCmRxB+SClMycWXPnt3c3cmwcJw4Vvxc8fuXGQgyw286A7MIIYQQM0ERJoQQQswERTgVSE7qSZMmxclNTThO/EylD/z+cZws4TPFOWFCCCHETNASJoQQQswERZgQQggxExRhQgghxExQhFPI3LlzUbx4cTg5OaF27do4cuSIaf8yFsKePXvQoUMHuLu7I0uWLNiwYYO5u5QhkVKdNWvWVAkC8uXLp8p0SrUwEpf58+ejcuXKag2nbFJ3fOvWrRym1zB9+nT1/fvggw84VvGYPHmyGhvDrWzZskgvKMIpYOXKlaoOskTRnThxAl5eXmjdujUePHhg+r9QJkdqQ8v4yE0LSZzdu3dj6NChOHToEHx8fBAZGYlWrVqp8SOxSEUaEZTjx4/j2LFjaNasGTp27Ijz589zmBLh6NGj+Omnn9TNC0mYChUqqHzPuk1K5aYbkjGLGEetWrU0Q4cO1bejo6M17u7ummnTpnEok0A+buvXr+cYJYMHDx6o8dq9ezfH6zXkypVL8/PPP3OcEiA4OFjj6emp8fHx0TRu3FgzcuRIjlM8Jk2apPHy8tKYC1rCRhIREaHuwlu0aKHfJzmopX3w4EFT3yMRK0XS5glubm7m7kqGJTo6GitWrFDeAnFLk1cR70r79u3j/F6RV7ly5YqaMitRogR69+4NPz8/pBdWV0UptTx8+FB9+aUAhCHSvnTpktn6RSwHSf4uc3f169dHxYoVzd2dDMfZs2eV6IaFhcHFxQXr169XSfdJXOQGRabLxB1NEkdiepYuXYoyZcooV/Tnn3+Ohg0b4ty5c+lSmIciTEgGtF7kByBd56UyEfJjeerUKeUtWLNmDby9vdWcOoU4Fn9/f4wcOVLFF0jwKEmctm3b6h/LvLmIcrFixbBq1SoMGDAAaQ1F2Ejy5MkDW1tbfV1iHdIuUKCAKf82xAoZNmwY/v77bxVVLkFI5FUcHBxQqlQp9bh69erK0ps1a5YKPiJaZMpMAkWrVaumHxLx4Mnnas6cOQgPD1e/Y+RVcubMidKlS+Pq1atIDzgnnIIfAPni79ixI477UNqclyIpReLWRIDFtbpz5054eHhwMJOJfP9EVEgszZs3V2578Rjotho1aqj5TnlMAU6c58+f49q1ayhYsCDSA1rCKUCWJ4kLTD7UtWrVwg8//KCCQ/r162f6v5AFfKAN7yhv3LihfgQk4Kho0aJm7VtGc0EvW7YMGzduVPNQAQEBar/UNs2aNau5u5dhGDt2rHIfymdHCrDLmO3atQvbt283d9cyFPIZih9PkC1bNuTOnZtxBvEYPXq0ymUgLui7d++qpadyk9KrVy+kBxThFNCjRw8EBgZi4sSJ6seySpUq2LZt2yvBWgRqLWfTpk3j3MAIchMjwRAkNgmF0KRJkzhDsmTJErz77rscppeIi7Vv374qgEZuUGQOTwS4ZcuWHCOSIm7fvq0E99GjR8ibNy8aNGig1uvL4/SAVZQIIYQQM8E5YUIIIcRMUIQJIYQQM0ERJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCiMnIkiULNmzYwBElJJlQhAmxECS9pYhg/K1Nmzbm7hohJBGYO5oQC0IEV/JNG+Lo6Gi2/hBCkoaWMCEWhAiu1LU23HLlyqWeE6tYCkVIFSKpzFSiRAmsWbMmzuul/F2zZs3U81JxZ9CgQaoSliGLFy9GhQoV1LWk3JuUYDTk4cOH6Ny5M5ydneHp6YlNmzbpn3vy5IkqpyfJ8eUa8nz8mwZCrAmKMCFWxIQJE9ClSxecPn1aiWHPnj1x8eJF9ZyU42zdurUS7aNHj2L16tX4999/44isiLiUXRRxFsEWgS1VqlSca3z++efo3r07zpw5g3bt2qnrPH78WH/9CxcuYOvWreq6cr48efKk8ygQkoHQEEIsAm9vb42tra0mW7ZscbapU6eq5+XrPnjw4DivqV27tmbIkCHq8cKFCzW5cuXSPH/+XP/85s2bNTY2NpqAgADVdnd313z22WeJ9kGuMX78eH1bziX7tm7dqtodOnTQ9OvXz8TvnJDMC+eECbEgpHazrjaxDjc3N/3junXrxnlO2qdOnVKPxTL18vJSxd911K9fHzExMfD19VXubCl63rx58yT7IDV+dci5smfPruoAC0OGDFGW+IkTJ9CqVSt06tQJ9erVS+W7JiTzQhEmxIIQ0YvvHjYVMoebHOzt7eO0RbxFyAWZj7516xa2bNkCHx8fJeji3v7uu+/SpM+EZHQ4J0yIFXHo0KFX2uXKlVOP5X+ZK5a5YR379++HjY0NypQpA1dXVxQvXhw7duxIVR8kKMvb2xt//PEHfvjhByxcuDBV5yMkM0NLmBALIjw8HAEBAXH22dnZ6YOfJNiqRo0aaNCgAf78808cOXIEv/zyi3pOAqgmTZqkBHLy5MkIDAzE8OHD8c477yB//vzqGNk/ePBg5MuXT1m1wcHBSqjluOQwceJEVK9eXUVXS1///vtv/U0AIdYIRZgQC2Lbtm1q2ZAhYsVeunRJH7m8YsUKvP/+++q45cuXo3z58uo5WVK0fft2jBw5EjVr1lRtmb+dMWOG/lwi0GFhYZg5cyZGjx6txL1r167J7p+DgwPGjh2LmzdvKvd2w4YNVX8IsVaySHSWuTtBCEl7ZG52/fr1KhiKEJIx4JwwIYQQYiYowoQQQoiZ4JwwIVYCZ54IyXjQEiaEEELMBEWYEEIIMRMUYUIIIcRMUIQJIYQQM0ERJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCYB7+D6azCLb5uepmAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90aba699-21bc-42de-a69c-99f370bb0363",
   "metadata": {},
   "source": [
    "- Based on the accuracy plot above, we can see that the model achieves a relatively high training and validation accuracy after epochs 4 and 5\n",
    "- However, we have to keep in mind that we specified `eval_iter=5` in the training function earlier, which means that we only estimated the training and validation set performances\n",
    "- We can compute the training, validation, and test set performances over the complete dataset as follows below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "UHWaJFrjY0zW",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UHWaJFrjY0zW",
    "outputId": "e111e6e6-b147-4159-eb9d-19d4e809ed34"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 97.21%\n",
      "Validation accuracy: 97.32%\n",
      "Test accuracy: 95.67%\n"
     ]
    }
   ],
   "source": [
    "train_accuracy = calc_accuracy_loader(train_loader, model, device)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6882649f-dc7b-401f-84d2-024ff79c74a1",
   "metadata": {},
   "source": [
    "- We can see that the training and validation set performances are practically identical\n",
    "- However, based on the slightly lower test set performance, we can see that the model overfits the training data to a very small degree, as well as the validation data that has been used for tweaking some of the hyperparameters, such as the learning rate\n",
    "- This is normal, however, and this gap could potentially be further reduced by increasing the model's dropout rate (`drop_rate`) or the `weight_decay` in the optimizer setting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74d9ad7-3ec1-450e-8c9f-4fc46d3d5bb0",
   "metadata": {},
   "source": [
    "## 6.8 Using the LLM as a spam classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72ebcfa2-479e-408b-9cf0-7421f6144855",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/18.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5408e6-83e4-4e5a-8503-c2fba6073f31",
   "metadata": {},
   "source": [
    "- Finally, let's use the finetuned GPT model in action\n",
    "- The `classify_review` function below implements the data preprocessing steps similar to the `SpamDataset` we implemented earlier\n",
    "- Then, the function returns the predicted integer class label from the model and returns the corresponding class name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "aHdn6xvL-IW5",
   "metadata": {
    "id": "aHdn6xvL-IW5"
   },
   "outputs": [],
   "source": [
    "def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):\n",
    "    model.eval()\n",
    "\n",
    "    # Prepare inputs to the model\n",
    "    input_ids = tokenizer.encode(text)\n",
    "    supported_context_length = model.pos_emb.weight.shape[0]\n",
    "    # Note: In the book, this was originally written as pos_emb.weight.shape[1] by mistake\n",
    "    # It didn't break the code but would have caused unnecessary truncation (to 768 instead of 1024)\n",
    "\n",
    "    # Truncate sequences if they too long\n",
    "    input_ids = input_ids[:min(max_length, supported_context_length)]\n",
    "    assert max_length is not None, (\n",
    "        \"max_length must be specified. If you want to use the full model context, \"\n",
    "        \"pass max_length=model.pos_emb.weight.shape[0].\"\n",
    "    )\n",
    "    assert max_length <= supported_context_length, (\n",
    "        f\"max_length ({max_length}) exceeds model's supported context length ({supported_context_length}).\"\n",
    "    )    \n",
    "    # Alternatively, a more robust version is the following one, which handles the max_length=None case better\n",
    "    # max_len = min(max_length,supported_context_length) if max_length else supported_context_length\n",
    "    # input_ids = input_ids[:max_len]\n",
    "    \n",
    "    # Pad sequences to the longest sequence\n",
    "    input_ids += [pad_token_id] * (max_length - len(input_ids))\n",
    "    input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) # add batch dimension\n",
    "\n",
    "    # Model inference\n",
    "    with torch.no_grad():\n",
    "        logits = model(input_tensor)[:, -1, :]  # Logits of the last output token\n",
    "    predicted_label = torch.argmax(logits, dim=-1).item()\n",
    "\n",
    "    # Return the classified result\n",
    "    return \"spam\" if predicted_label == 1 else \"not spam\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29682d8-a899-4d9b-b973-f8d5ec68172c",
   "metadata": {},
   "source": [
    "- Let's try it out on a few examples below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "apU_pf51AWSV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "apU_pf51AWSV",
    "outputId": "d0fde0a5-e7a3-4dbe-d9c5-0567dbab7e62"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spam\n"
     ]
    }
   ],
   "source": [
    "text_1 = (\n",
    "    \"You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_1, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "1g5VTOo_Ajs5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1g5VTOo_Ajs5",
    "outputId": "659b08eb-b6a9-4a8a-9af7-d94c757e93c2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not spam\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Hey, just wanted to check if we're still on\"\n",
    "    \" for dinner tonight? Let me know!\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_2, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf736e39-0d47-40c1-8d18-1f716cf7a81e",
   "metadata": {},
   "source": [
    "- Finally, let's save the model in case we want to reuse the model later without having to train it again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "mYnX-gI1CfQY",
   "metadata": {
    "id": "mYnX-gI1CfQY"
   },
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), \"review_classifier.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba78cf7c-6b80-4f71-a50e-3ccc73839af6",
   "metadata": {},
   "source": [
    "- Then, in a new session, we could load the model as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "cc4e68a5-d492-493b-87ef-45c475f353f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_state_dict = torch.load(\"review_classifier.pth\", map_location=device, weights_only=True)\n",
    "model.load_state_dict(model_state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4",
   "metadata": {
    "id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4"
   },
   "source": [
    "## Summary and takeaways"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafdc910-d616-47ab-aa85-f90c6e7ed80e",
   "metadata": {},
   "source": [
    "- See the [./gpt_class_finetune.py](./gpt_class_finetune.py) script, a self-contained script for classification finetuning\n",
    "- You can find the exercise solutions in [./exercise-solutions.ipynb](./exercise-solutions.ipynb)\n",
    "- In addition, interested readers can find an introduction to parameter-efficient training with low-rank adaptation (LoRA) in [appendix E](../../appendix-E)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "V100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
